[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:46:34 dist-test-slave-hvcl krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:46:34 dist-test-slave-hvcl krb5kdc[399](info): set up 2 sockets
Sep 24 04:46:34 dist-test-slave-hvcl krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 04:46:35.659192 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:46:35 dist-test-slave-hvcl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:46:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:46:35.778450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37543
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:35.887917 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:35.888208 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:35.888266 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:35.892318 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:46:35.892390 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:35.892418 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:35.892438 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:46:35.892459 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:46:35.897783 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:35.898877 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:35.900005 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:46:35.909320 412 server_base.cc:1034] running on GCE node
W20260924 04:46:35.909448 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:35.909454 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:35.909611 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:35.910264 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:35.911489 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:35.912667 412 hybrid_clock.cc:648] HybridClock initialized: now 1790225195912604 us; error 76 us; skew 500 ppm
Sep 24 04:46:35 dist-test-slave-hvcl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:35.915555 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:46:35.916430 412 webserver.cc:460] Webserver started at http://127.0.98.62:43567/ using document root <none> and password file <none>
I20260924 04:46:35.917016 412 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:35.917083 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:35.917289 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:35.919406 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "fbf8c5e9c6c740d3b3b16a1329d38633"
format_stamp: "Formatted at 2026-09-24 04:46:35 on dist-test-slave-hvcl"
server_key: "74f6373b999cf25aeb9a78218421743b"
server_key_iv: "7b18161cdea3696cc275d87581d55f1f"
server_key_version: "encryptionkey@0"
I20260924 04:46:35.919955 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "fbf8c5e9c6c740d3b3b16a1329d38633"
format_stamp: "Formatted at 2026-09-24 04:46:35 on dist-test-slave-hvcl"
server_key: "74f6373b999cf25aeb9a78218421743b"
server_key_iv: "7b18161cdea3696cc275d87581d55f1f"
server_key_version: "encryptionkey@0"
I20260924 04:46:35.923702 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:46:35.926295 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:35.927634 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:46:35.927757 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "fbf8c5e9c6c740d3b3b16a1329d38633"
format_stamp: "Formatted at 2026-09-24 04:46:35 on dist-test-slave-hvcl"
server_key: "74f6373b999cf25aeb9a78218421743b"
server_key_iv: "7b18161cdea3696cc275d87581d55f1f"
server_key_version: "encryptionkey@0"
I20260924 04:46:35.927865 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:35.940286 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:35.943475 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:35.943662 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:35.951069 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37543
I20260924 04:46:35.951115 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37543 every 8 connection(s)
I20260924 04:46:35.952064 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:46:35.955062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 412
I20260924 04:46:35.955160 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:35.955232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:46:35.957052 392 external_mini_cluster.cc:1428] Setting key 5edc1d11b3b6d870c1b0520bae0b5e11
I20260924 04:46:35.962149 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633: Bootstrap starting.
I20260924 04:46:35.964443 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:35.965163 480 log.cc:824] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:35.967406 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633: No bootstrap required, opened a new log
I20260924 04:46:35.971354 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf8c5e9c6c740d3b3b16a1329d38633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37543 } }
I20260924 04:46:35.971597 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:35.971634 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf8c5e9c6c740d3b3b16a1329d38633, State: Initialized, Role: FOLLOWER
I20260924 04:46:35.972081 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf8c5e9c6c740d3b3b16a1329d38633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37543 } }
I20260924 04:46:35.972213 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:46:35.972275 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:46:35.972368 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:35.973273 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf8c5e9c6c740d3b3b16a1329d38633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37543 } }
I20260924 04:46:35.973601 480 leader_election.cc:302] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbf8c5e9c6c740d3b3b16a1329d38633; no voters:
I20260924 04:46:35.973903 480 leader_election.cc:288] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:46:35.974028 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:35.974207 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [term 1 LEADER]: Becoming Leader. State: Replica: fbf8c5e9c6c740d3b3b16a1329d38633, State: Running, Role: LEADER
I20260924 04:46:35.974583 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf8c5e9c6c740d3b3b16a1329d38633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37543 } }
I20260924 04:46:35.975313 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 04:46:35 dist-test-slave-hvcl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:35.976284 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbf8c5e9c6c740d3b3b16a1329d38633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf8c5e9c6c740d3b3b16a1329d38633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37543 } } }
I20260924 04:46:35.976405 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:35.976714 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbf8c5e9c6c740d3b3b16a1329d38633. Latest consensus state: current_term: 1 leader_uuid: "fbf8c5e9c6c740d3b3b16a1329d38633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf8c5e9c6c740d3b3b16a1329d38633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37543 } } }
I20260924 04:46:35.976807 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633 [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:35.977866 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:46:35.980923 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:46:35.982676 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:35.960980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.0.98.62:37543)
0924 04:46:35.961492 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:35.961511 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:35.962748 (+ 1237us) server_negotiation.cc:407] Connection header received
0924 04:46:35.965957 (+ 3209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:35.965979 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:35.966384 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:35.966755 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:35.969060 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:35.970551 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:35.971916 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:35.972453 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:35.972625 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:35.977168 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:35.977176 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:35.977193 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:35.977230 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:35.979186 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:35.979724 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:35.979731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:35.979735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:35.979856 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:35.980197 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:35.980201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:35.980202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:35.980582 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:35.980758 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:35.981352 (+ 594us) server_negotiation.cc:299] Negotiation successful
0924 04:46:35.981611 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":156,"threads_started":1}
I20260924 04:46:35.987216 494 catalog_manager.cc:1347] Generated new cluster ID: 1fc4c506416c4fca9269e57afa6acf17
I20260924 04:46:35.987288 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:46:36.001353 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:46:36.002370 494 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:46:36.012328 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbf8c5e9c6c740d3b3b16a1329d38633: Generated new TSK 0
I20260924 04:46:36.013084 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:46:36.042716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37543
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:36.155887 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:36.156157 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:36.156199 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:36.160363 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:36.160430 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:36.160497 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:46:36.165882 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:36.167009 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:36.168105 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:46:36.178112 506 server_base.cc:1034] running on GCE node
W20260924 04:46:36.178071 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:36.178099 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:36.178071 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:36.178735 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:36.179425 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:36.180588 506 hybrid_clock.cc:648] HybridClock initialized: now 1790225196180572 us; error 30 us; skew 500 ppm
Sep 24 04:46:36 dist-test-slave-hvcl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:36.189162 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:46:36.190083 506 webserver.cc:460] Webserver started at http://127.0.98.1:39987/ using document root <none> and password file <none>
I20260924 04:46:36.190653 506 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:36.190712 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:36.190893 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:36.192956 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "2ece6b1210514d8f90498c68fde13ece"
format_stamp: "Formatted at 2026-09-24 04:46:36 on dist-test-slave-hvcl"
server_key: "1f40c3ebb2212be6802f6ab74d88d0e2"
server_key_iv: "91736eab7b2e037b82c340a041f8dd46"
server_key_version: "encryptionkey@0"
I20260924 04:46:36.193500 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2ece6b1210514d8f90498c68fde13ece"
format_stamp: "Formatted at 2026-09-24 04:46:36 on dist-test-slave-hvcl"
server_key: "1f40c3ebb2212be6802f6ab74d88d0e2"
server_key_iv: "91736eab7b2e037b82c340a041f8dd46"
server_key_version: "encryptionkey@0"
I20260924 04:46:36.197696 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:46:36.201289 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:36.203164 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:46:36.203302 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "2ece6b1210514d8f90498c68fde13ece"
format_stamp: "Formatted at 2026-09-24 04:46:36 on dist-test-slave-hvcl"
server_key: "1f40c3ebb2212be6802f6ab74d88d0e2"
server_key_iv: "91736eab7b2e037b82c340a041f8dd46"
server_key_version: "encryptionkey@0"
I20260924 04:46:36.203423 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:36.221601 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:36.225797 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:36.226019 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:36.226778 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:36.227881 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:36.227944 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:36.228009 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:36.228041 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:36.238905 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43157
I20260924 04:46:36.238950 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43157 every 8 connection(s)
I20260924 04:46:36.239893 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:46:36.249035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 506
I20260924 04:46:36.249137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:46:36.249369 392 external_mini_cluster.cc:1428] Setting key 356ae9c1980b01ccaa05409d67a2fac8
Sep 24 04:46:36 dist-test-slave-hvcl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:36.253854 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:36.242010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43701 (local address 127.0.98.62:37543)
0924 04:46:36.242156 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:36.242159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:36.243035 (+ 876us) server_negotiation.cc:407] Connection header received
0924 04:46:36.244371 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:36.244375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:36.244431 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:36.244504 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:36.245687 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.246215 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.247440 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.247671 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.247760 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:36.250175 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:36.250179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:36.250181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:36.250205 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:36.251584 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:36.252106 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:36.252110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:36.252112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:36.252215 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:36.252578 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:36.252580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:36.252581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:36.252767 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:36.252840 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:36.253532 (+ 692us) server_negotiation.cc:299] Negotiation successful
0924 04:46:36.253688 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:46:36.254616 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:36.242427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37543 (local address 127.0.98.1:43701)
0924 04:46:36.242874 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:36.242917 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:36.244159 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:36.244695 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:36.244702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:36.245060 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:36.245538 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:36.245550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.246327 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.246330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:36.247329 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:36.247338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.247814 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.247818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:36.247992 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:36.248551 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:36.248567 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:36.250003 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:36.251695 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:36.251701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:36.251718 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:36.251979 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:36.252301 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:36.252303 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:36.252305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:36.252496 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:36.252857 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:36.252864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:36.253188 (+ 324us) client_negotiation.cc:769] Sending connection context
0924 04:46:36.253374 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 04:46:36.253660 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":119,"threads_started":1}
I20260924 04:46:36.255882 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37543
I20260924 04:46:36.256132 635 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:36.256628 635 heartbeater.cc:511] Master 127.0.98.62:37543 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:46:36.258411 444 ts_manager.cc:194] Registered new tserver with Master: 2ece6b1210514d8f90498c68fde13ece (127.0.98.1:43157)
I20260924 04:46:36.259693 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:46:36.277958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37543
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:36.389312 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:36.389590 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:36.389658 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:36.393697 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:36.393764 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:36.393846 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:46:36.399156 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:36.400238 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:36.401268 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:36.411430 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:36.411433 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:36.411494 640 server_base.cc:1034] running on GCE node
W20260924 04:46:36.411433 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:36.412256 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:36.412874 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:36.414083 640 hybrid_clock.cc:648] HybridClock initialized: now 1790225196414053 us; error 47 us; skew 500 ppm
Sep 24 04:46:36 dist-test-slave-hvcl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:36.416895 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:46:36.417824 640 webserver.cc:460] Webserver started at http://127.0.98.2:37513/ using document root <none> and password file <none>
I20260924 04:46:36.418350 640 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:36.418407 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:36.418613 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:36.420639 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "ba5231710c944032a5dcf44a5e7e254c"
format_stamp: "Formatted at 2026-09-24 04:46:36 on dist-test-slave-hvcl"
server_key: "ce116b1a79919b8b3c37615a7144c103"
server_key_iv: "926c6980c441da40c40d14a1fd1f6141"
server_key_version: "encryptionkey@0"
I20260924 04:46:36.421154 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ba5231710c944032a5dcf44a5e7e254c"
format_stamp: "Formatted at 2026-09-24 04:46:36 on dist-test-slave-hvcl"
server_key: "ce116b1a79919b8b3c37615a7144c103"
server_key_iv: "926c6980c441da40c40d14a1fd1f6141"
server_key_version: "encryptionkey@0"
I20260924 04:46:36.424979 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 04:46:36.427768 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:36.429215 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 04:46:36.429370 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "ba5231710c944032a5dcf44a5e7e254c"
format_stamp: "Formatted at 2026-09-24 04:46:36 on dist-test-slave-hvcl"
server_key: "ce116b1a79919b8b3c37615a7144c103"
server_key_iv: "926c6980c441da40c40d14a1fd1f6141"
server_key_version: "encryptionkey@0"
I20260924 04:46:36.429513 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:36.452275 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:36.455493 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:36.455655 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:36.456184 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:36.456947 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:36.456990 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:36.457028 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:36.457048 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:36.468267 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44727
I20260924 04:46:36.468304 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44727 every 8 connection(s)
I20260924 04:46:36.469416 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:46:36.474202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 640
I20260924 04:46:36.474314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:46:36.474537 392 external_mini_cluster.cc:1428] Setting key e43b413053bbb1a1161d4b705b6eeb29
Sep 24 04:46:36 dist-test-slave-hvcl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:46:36.484469 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:36.471327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57437 (local address 127.0.98.62:37543)
0924 04:46:36.471446 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:36.471449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:36.472561 (+ 1112us) server_negotiation.cc:407] Connection header received
0924 04:46:36.473930 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:36.473933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:36.474013 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:36.474083 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:36.475439 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.476073 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.477470 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.477768 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.477854 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:36.480205 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:36.480209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:36.480212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:36.480240 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:36.481989 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:36.482627 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:36.482631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:36.482633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:36.482720 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:36.483197 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:36.483201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:36.483203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:36.483443 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:36.483553 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:36.484154 (+ 601us) server_negotiation.cc:299] Negotiation successful
0924 04:46:36.484287 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:46:36.485327 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:36.471916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37543 (local address 127.0.98.2:57437)
0924 04:46:36.472413 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:36.472455 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:36.473719 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:36.474266 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:36.474273 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:36.474674 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:36.475291 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:36.475308 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.476186 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.476189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:36.477359 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:36.477368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.477902 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.477906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:36.478076 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:36.478619 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:36.478634 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:36.480041 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:36.482110 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:36.482117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:36.482135 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:36.482505 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:36.482835 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:36.482838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:36.482840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:36.483082 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:36.483556 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:36.483563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:36.483877 (+ 314us) client_negotiation.cc:769] Sending connection context
0924 04:46:36.484057 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 04:46:36.484326 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":160,"threads_started":1}
I20260924 04:46:36.486653 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:46:36.486943 769 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:36.487483 769 heartbeater.cc:511] Master 127.0.98.62:37543 requested a full tablet report, sending...
I20260924 04:46:36.488584 444 ts_manager.cc:194] Registered new tserver with Master: ba5231710c944032a5dcf44a5e7e254c (127.0.98.2:44727)
I20260924 04:46:36.489336 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:46:36.502434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37543
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:36.612165 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:36.612481 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:36.612555 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:36.616670 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:36.616732 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:36.616813 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:46:36.622166 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:36.623262 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:36.624280 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:46:36.634061 774 server_base.cc:1034] running on GCE node
W20260924 04:46:36.634044 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:36.634039 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:36.634292 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:36.634694 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:36.635383 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:36.636581 774 hybrid_clock.cc:648] HybridClock initialized: now 1790225196636564 us; error 28 us; skew 500 ppm
Sep 24 04:46:36 dist-test-slave-hvcl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:36.639356 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:46:36.640234 774 webserver.cc:460] Webserver started at http://127.0.98.3:37213/ using document root <none> and password file <none>
I20260924 04:46:36.640772 774 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:36.640834 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:36.641027 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:36.643097 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "cfdd08cda2cf4c7fbf7d072ed26a98b4"
format_stamp: "Formatted at 2026-09-24 04:46:36 on dist-test-slave-hvcl"
server_key: "4550b9e16747cf878295e9e4de723dc9"
server_key_iv: "d92ac803ad98f17e92a2ba5c495c7cd6"
server_key_version: "encryptionkey@0"
I20260924 04:46:36.643644 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cfdd08cda2cf4c7fbf7d072ed26a98b4"
format_stamp: "Formatted at 2026-09-24 04:46:36 on dist-test-slave-hvcl"
server_key: "4550b9e16747cf878295e9e4de723dc9"
server_key_iv: "d92ac803ad98f17e92a2ba5c495c7cd6"
server_key_version: "encryptionkey@0"
I20260924 04:46:36.647388 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:46:36.649963 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:36.651273 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:46:36.651382 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "cfdd08cda2cf4c7fbf7d072ed26a98b4"
format_stamp: "Formatted at 2026-09-24 04:46:36 on dist-test-slave-hvcl"
server_key: "4550b9e16747cf878295e9e4de723dc9"
server_key_iv: "d92ac803ad98f17e92a2ba5c495c7cd6"
server_key_version: "encryptionkey@0"
I20260924 04:46:36.651479 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:36.663380 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:36.666260 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:36.666425 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:36.667032 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:36.667836 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:36.667881 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:36.667932 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:36.667960 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:36.677022 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43311
I20260924 04:46:36.677047 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43311 every 8 connection(s)
I20260924 04:46:36.677942 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:46:36.678505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 774
I20260924 04:46:36.678638 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:46:36.679021 392 external_mini_cluster.cc:1428] Setting key 6f7a93cb4d6de5ada8bfc3cef45817e3
Sep 24 04:46:36 dist-test-slave-hvcl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:36.691825 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:36.680174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40265 (local address 127.0.98.62:37543)
0924 04:46:36.680303 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:36.680307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:36.681494 (+ 1187us) server_negotiation.cc:407] Connection header received
0924 04:46:36.682857 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:36.682861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:36.682913 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:36.683010 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:36.684228 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.684780 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.685931 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.686168 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.686259 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:36.688634 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:36.688638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:36.688641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:36.688668 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:36.689882 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:36.690357 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:36.690360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:36.690362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:36.690438 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:36.690743 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:36.690746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:36.690748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:36.690926 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:36.691039 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:36.691529 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 04:46:36.691649 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:46:36.692766 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:36.680826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37543 (local address 127.0.98.3:40265)
0924 04:46:36.681349 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:36.681407 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:36.682650 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:36.683186 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:36.683201 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:36.683573 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:36.684088 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:36.684103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.684898 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.684900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:36.685784 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:36.685794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.686307 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.686310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:36.686470 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:36.687078 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:36.687094 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:36.688471 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:36.689987 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:36.689993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:36.690009 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:36.690265 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:36.690517 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:36.690520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:36.690522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:36.690653 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:36.691034 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:36.691040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:36.691302 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 04:46:36.691472 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 04:46:36.691807 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":126,"threads_started":1}
I20260924 04:46:36.693897 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37543
I20260924 04:46:36.694139 903 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:36.694625 903 heartbeater.cc:511] Master 127.0.98.62:37543 requested a full tablet report, sending...
I20260924 04:46:36.695597 444 ts_manager.cc:194] Registered new tserver with Master: cfdd08cda2cf4c7fbf7d072ed26a98b4 (127.0.98.3:43311)
I20260924 04:46:36.696184 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40265
I20260924 04:46:36.705758 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:46:36 dist-test-slave-hvcl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225196, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:46:36 dist-test-slave-hvcl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225196, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:46:36.731637 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:36.722164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.0.98.1:43157)
0924 04:46:36.722449 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:36.722465 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:36.722491 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:46:36.722604 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:36.722611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:36.722806 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:36.722941 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:36.723655 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.724439 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.725265 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.725640 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.725734 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:36.728119 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:36.728128 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:36.728141 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:36.728175 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:36.729666 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:36.730243 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:36.730250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:36.730251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:36.730338 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:36.730605 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:36.730608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:36.730610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:36.730863 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:36.731118 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:36.731299 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 04:46:36.731451 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":63,"threads_started":1}
W20260924 04:46:36.732580 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35658
Sep 24 04:46:36 dist-test-slave-hvcl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:46:36 dist-test-slave-hvcl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:46:36.757215 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:36.749011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.0.98.1:43157)
0924 04:46:36.749166 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:36.749171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:36.749260 (+ 89us) server_negotiation.cc:407] Connection header received
0924 04:46:36.749393 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:36.749397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:36.749442 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:36.749507 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:36.750210 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.750732 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.751601 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:36.751837 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:36.751924 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:36.754227 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:36.754230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:36.754232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:36.754271 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:36.755703 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:36.756082 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:36.756086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:36.756087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:36.756203 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:36.756449 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:36.756452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:36.756454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:36.756619 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:36.756729 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:36.756947 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 04:46:36.757054 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:46:36.758212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 506
I20260924 04:46:36.763382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 640
I20260924 04:46:36.769739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 774
I20260924 04:46:36.774564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 412
2026-09-24T04:46:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2089 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:46:36 dist-test-slave-hvcl krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:46:36 dist-test-slave-hvcl krb5kdc[917](info): set up 2 sockets
Sep 24 04:46:36 dist-test-slave-hvcl krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 04:46:37.730369 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:46:37 dist-test-slave-hvcl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:46:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:46:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:46:37.853902 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45687
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:37.964542 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:37.964838 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:37.964895 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:37.969038 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:46:37.969117 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:37.969146 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:37.969165 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:46:37.969184 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:46:37.974623 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:37.975756 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:37.976804 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:37.985662 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:37.985666 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:37.985705 930 server_base.cc:1034] running on GCE node
W20260924 04:46:37.985662 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:37.986495 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:37.987664 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:37.988838 930 hybrid_clock.cc:648] HybridClock initialized: now 1790225197988819 us; error 33 us; skew 500 ppm
Sep 24 04:46:37 dist-test-slave-hvcl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:37.991606 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:46:37.992494 930 webserver.cc:460] Webserver started at http://127.0.98.62:39707/ using document root <none> and password file <none>
I20260924 04:46:37.993017 930 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:37.993086 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:37.993249 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:37.995329 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "ca3bb821af3347058bf1f18d2949e939"
format_stamp: "Formatted at 2026-09-24 04:46:37 on dist-test-slave-hvcl"
server_key: "5bcd5077691e1fbd00449c72a4d87924"
server_key_iv: "d9c528179853dde090aab25fb913aa0c"
server_key_version: "encryptionkey@0"
I20260924 04:46:37.995847 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca3bb821af3347058bf1f18d2949e939"
format_stamp: "Formatted at 2026-09-24 04:46:37 on dist-test-slave-hvcl"
server_key: "5bcd5077691e1fbd00449c72a4d87924"
server_key_iv: "d9c528179853dde090aab25fb913aa0c"
server_key_version: "encryptionkey@0"
I20260924 04:46:37.999552 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:46:38.001988 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:38.003312 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:46:38.003422 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "ca3bb821af3347058bf1f18d2949e939"
format_stamp: "Formatted at 2026-09-24 04:46:37 on dist-test-slave-hvcl"
server_key: "5bcd5077691e1fbd00449c72a4d87924"
server_key_iv: "d9c528179853dde090aab25fb913aa0c"
server_key_version: "encryptionkey@0"
I20260924 04:46:38.003539 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:38.019372 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:38.022626 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:38.022795 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:38.029678 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45687
I20260924 04:46:38.029738 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45687 every 8 connection(s)
I20260924 04:46:38.030666 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:46:38.033613 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:38.039393 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939: Bootstrap starting.
I20260924 04:46:38.040390 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 930
I20260924 04:46:38.040490 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:46:38.040721 392 external_mini_cluster.cc:1428] Setting key 71e77a5d433435972a6eb6588ef2530e
I20260924 04:46:38.042186 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:38.042955 998 log.cc:824] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:38.045217 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939: No bootstrap required, opened a new log
I20260924 04:46:38.048727 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3bb821af3347058bf1f18d2949e939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45687 } }
I20260924 04:46:38.048971 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:38.049031 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3bb821af3347058bf1f18d2949e939, State: Initialized, Role: FOLLOWER
Sep 24 04:46:38 dist-test-slave-hvcl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:38.049525 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3bb821af3347058bf1f18d2949e939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45687 } }
I20260924 04:46:38.049688 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:46:38.049762 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:46:38.049867 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:38.050813 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3bb821af3347058bf1f18d2949e939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45687 } }
I20260924 04:46:38.051172 998 leader_election.cc:302] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca3bb821af3347058bf1f18d2949e939; no voters:
I20260924 04:46:38.051471 998 leader_election.cc:288] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:46:38.051594 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:38.051810 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [term 1 LEADER]: Becoming Leader. State: Replica: ca3bb821af3347058bf1f18d2949e939, State: Running, Role: LEADER
I20260924 04:46:38.052139 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3bb821af3347058bf1f18d2949e939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45687 } }
I20260924 04:46:38.052687 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:46:38.054149 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca3bb821af3347058bf1f18d2949e939. Latest consensus state: current_term: 1 leader_uuid: "ca3bb821af3347058bf1f18d2949e939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3bb821af3347058bf1f18d2949e939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45687 } } }
I20260924 04:46:38.054277 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:38.054674 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:46:38.054642 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca3bb821af3347058bf1f18d2949e939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3bb821af3347058bf1f18d2949e939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45687 } } }
I20260924 04:46:38.054733 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939 [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:38.055511 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:38.041809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.0.98.62:45687)
0924 04:46:38.042257 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:38.042277 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:38.042507 (+ 230us) server_negotiation.cc:407] Connection header received
0924 04:46:38.043331 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:38.043352 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:38.043709 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:38.044042 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:38.044784 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:38.045800 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:38.046637 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:38.047182 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:38.047327 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:38.049837 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:38.049845 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:38.049863 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:38.049892 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:38.051762 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:38.052293 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:38.052299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:38.052305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:38.052509 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:38.052870 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:38.052873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:38.052875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:38.053440 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:38.053635 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:38.054107 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 04:46:38.054427 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":115,"threads_started":1}
I20260924 04:46:38.058486 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:46:38.063795 1012 catalog_manager.cc:1347] Generated new cluster ID: d1b3b2c37b174e9f86838cba9c1a12e0
I20260924 04:46:38.063869 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:46:38.099200 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:46:38.100214 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:46:38.104847 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca3bb821af3347058bf1f18d2949e939: Generated new TSK 0
I20260924 04:46:38.105690 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:46:38.143044 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45687
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:38.250948 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:38.251263 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:38.251329 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:38.255460 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:38.255520 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:38.255604 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:46:38.260809 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45687
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:38.261869 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:38.262898 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:38.272446 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:38.272454 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:38.272454 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:38.272571 1024 server_base.cc:1034] running on GCE node
I20260924 04:46:38.273070 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:38.273694 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:38.274857 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790225198274834 us; error 38 us; skew 500 ppm
Sep 24 04:46:38 dist-test-slave-hvcl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:38.277509 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:46:38.278365 1024 webserver.cc:460] Webserver started at http://127.0.98.1:41057/ using document root <none> and password file <none>
I20260924 04:46:38.278878 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:38.278941 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:38.279198 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:38.281296 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "554d11e0726f4f6bad8c56d5cd0bb17c"
format_stamp: "Formatted at 2026-09-24 04:46:38 on dist-test-slave-hvcl"
server_key: "9ccf3b1cb4b0481e5466521cba8af810"
server_key_iv: "158ed3846f09859a250f1ff7e0f5841a"
server_key_version: "encryptionkey@0"
I20260924 04:46:38.281814 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "554d11e0726f4f6bad8c56d5cd0bb17c"
format_stamp: "Formatted at 2026-09-24 04:46:38 on dist-test-slave-hvcl"
server_key: "9ccf3b1cb4b0481e5466521cba8af810"
server_key_iv: "158ed3846f09859a250f1ff7e0f5841a"
server_key_version: "encryptionkey@0"
I20260924 04:46:38.285529 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:46:38.287948 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:38.289150 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:46:38.289258 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "554d11e0726f4f6bad8c56d5cd0bb17c"
format_stamp: "Formatted at 2026-09-24 04:46:38 on dist-test-slave-hvcl"
server_key: "9ccf3b1cb4b0481e5466521cba8af810"
server_key_iv: "158ed3846f09859a250f1ff7e0f5841a"
server_key_version: "encryptionkey@0"
I20260924 04:46:38.289354 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:38.309626 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:38.312911 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:38.313093 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:38.313687 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:38.314509 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:38.314555 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:38.314606 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:38.314634 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:38.322696 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36497
I20260924 04:46:38.322739 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36497 every 8 connection(s)
I20260924 04:46:38.323611 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:46:38.329396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 1024
I20260924 04:46:38.329499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:46:38.329783 392 external_mini_cluster.cc:1428] Setting key b6e511369e9a62347e4c783690a0d23a
Sep 24 04:46:38 dist-test-slave-hvcl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:38.338074 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:38.325467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35769 (local address 127.0.98.62:45687)
0924 04:46:38.325589 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:38.325593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:38.326482 (+ 889us) server_negotiation.cc:407] Connection header received
0924 04:46:38.327800 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:38.327803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:38.327855 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:38.327929 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:38.329118 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:38.329887 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:38.331498 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:38.331868 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:38.331973 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:38.334696 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:38.334700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:38.334703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:38.334730 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:38.336102 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:38.336590 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:38.336594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:38.336596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:38.336681 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:38.336996 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:38.337000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:38.337002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:38.337189 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:38.337276 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:38.337780 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 04:46:38.337908 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:46:38.338945 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:38.325870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45687 (local address 127.0.98.1:35769)
0924 04:46:38.326333 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:38.326376 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:38.327593 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:38.328066 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:38.328073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:38.328457 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:38.328927 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:38.328940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:38.330007 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:38.330011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:38.331378 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:38.331390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:38.332011 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:38.332015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:38.332225 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:38.332843 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:38.332864 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:38.334521 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:38.336215 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:38.336221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:38.336238 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:38.336496 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:38.336766 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:38.336769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:38.336771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:38.336912 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:38.337285 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:38.337291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:38.337551 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 04:46:38.337721 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 04:46:38.338060 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":131,"threads_started":1}
I20260924 04:46:38.340150 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45687
I20260924 04:46:38.340386 1153 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:38.340873 1153 heartbeater.cc:511] Master 127.0.98.62:45687 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:46:38.342396 962 ts_manager.cc:194] Registered new tserver with Master: 554d11e0726f4f6bad8c56d5cd0bb17c (127.0.98.1:36497)
I20260924 04:46:38.343566 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:46:38.359258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45687
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:38.474272 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:38.474561 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:38.474607 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:38.478717 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:38.478782 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:38.478843 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:46:38.484316 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45687
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:38.485430 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:38.486472 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:38.496438 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:38.496464 1158 server_base.cc:1034] running on GCE node
W20260924 04:46:38.496486 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:38.496457 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:38.497103 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:38.497763 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:38.498946 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790225198498921 us; error 39 us; skew 500 ppm
Sep 24 04:46:38 dist-test-slave-hvcl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:38.502609 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:46:38.503672 1158 webserver.cc:460] Webserver started at http://127.0.98.2:34303/ using document root <none> and password file <none>
I20260924 04:46:38.504290 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:38.504356 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:38.504566 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:38.507196 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "7f734556919a45038b38f8cca47d5541"
format_stamp: "Formatted at 2026-09-24 04:46:38 on dist-test-slave-hvcl"
server_key: "b5de12e4312fed5a3ee63e61596d8975"
server_key_iv: "b8474368c68c20d7bf50e47259a6f598"
server_key_version: "encryptionkey@0"
I20260924 04:46:38.507820 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7f734556919a45038b38f8cca47d5541"
format_stamp: "Formatted at 2026-09-24 04:46:38 on dist-test-slave-hvcl"
server_key: "b5de12e4312fed5a3ee63e61596d8975"
server_key_iv: "b8474368c68c20d7bf50e47259a6f598"
server_key_version: "encryptionkey@0"
I20260924 04:46:38.511746 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:46:38.514500 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:38.515919 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:46:38.516036 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "7f734556919a45038b38f8cca47d5541"
format_stamp: "Formatted at 2026-09-24 04:46:38 on dist-test-slave-hvcl"
server_key: "b5de12e4312fed5a3ee63e61596d8975"
server_key_iv: "b8474368c68c20d7bf50e47259a6f598"
server_key_version: "encryptionkey@0"
I20260924 04:46:38.516146 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:38.527550 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:38.530791 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:38.531006 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:38.531621 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:38.532460 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:38.532505 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:38.532557 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:38.532590 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:38.541607 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38187
I20260924 04:46:38.541635 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38187 every 8 connection(s)
I20260924 04:46:38.542547 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:46:38.545660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 1158
I20260924 04:46:38.545789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:46:38.546034 392 external_mini_cluster.cc:1428] Setting key 9ff438ce1b05c77014cc144b7347a35f
Sep 24 04:46:38 dist-test-slave-hvcl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:46:38.559883 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:38.544588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57385 (local address 127.0.98.62:45687)
0924 04:46:38.544733 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:38.544737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:38.545563 (+ 826us) server_negotiation.cc:407] Connection header received
0924 04:46:38.547395 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:38.547397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:38.547440 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:38.547499 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:38.549853 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:38.550448 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:38.552273 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:38.552525 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:38.552622 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:38.555992 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:38.555995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:38.555997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:38.556025 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:38.557344 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:38.558031 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:38.558034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:38.558036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:38.558126 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:38.558631 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:38.558634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:38.558635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:38.558898 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:38.559034 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:38.559636 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 04:46:38.559755 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:46:38.560714 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:38.544900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45687 (local address 127.0.98.2:57385)
0924 04:46:38.545382 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:38.545427 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:38.547059 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:38.547734 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:38.547745 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:38.549133 (+ 1388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:38.549680 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:38.549695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:38.550596 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:38.550600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:38.552074 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:38.552088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:38.552703 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:38.552708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:38.552955 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:38.553721 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:38.553744 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:38.555791 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:38.557488 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:38.557494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:38.557508 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:38.557797 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:38.558330 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:38.558333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:38.558334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:38.558492 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:38.559069 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:38.559075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:38.559367 (+ 292us) client_negotiation.cc:769] Sending connection context
0924 04:46:38.559545 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 04:46:38.559843 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":125,"threads_started":1}
I20260924 04:46:38.561827 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45687
I20260924 04:46:38.562083 1287 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:38.562600 1287 heartbeater.cc:511] Master 127.0.98.62:45687 requested a full tablet report, sending...
I20260924 04:46:38.563755 962 ts_manager.cc:194] Registered new tserver with Master: 7f734556919a45038b38f8cca47d5541 (127.0.98.2:38187)
I20260924 04:46:38.564313 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:46:39.346029 1153 heartbeater.cc:503] Master 127.0.98.62:45687 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260924 04:46:39.469276 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.922s user 0.001s sys 0.003s
I20260924 04:46:39.469779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45687
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:46:39.566346 1287 heartbeater.cc:503] Master 127.0.98.62:45687 was elected leader, sending a full tablet report...
W20260924 04:46:39.584617 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:39.584874 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:39.584918 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:39.588990 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:39.589061 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:39.589121 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:46:39.594528 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45687
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:39.595636 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:39.596606 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:39.606182 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:39.606220 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:39.606251 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:39.606580 1292 server_base.cc:1034] running on GCE node
I20260924 04:46:39.607035 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:39.607725 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:39.608908 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790225199608880 us; error 45 us; skew 500 ppm
Sep 24 04:46:39 dist-test-slave-hvcl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:39.611620 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:46:39.612488 1292 webserver.cc:460] Webserver started at http://127.0.98.3:38113/ using document root <none> and password file <none>
I20260924 04:46:39.613045 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:39.613109 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:39.613303 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:39.615418 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "7da36394b3a144ceaeabb65a8ee94446"
format_stamp: "Formatted at 2026-09-24 04:46:39 on dist-test-slave-hvcl"
server_key: "763a7159495bcfa4a81cb09883d66937"
server_key_iv: "b6ee8ff6392e319e93e47da465461297"
server_key_version: "encryptionkey@0"
I20260924 04:46:39.615988 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7da36394b3a144ceaeabb65a8ee94446"
format_stamp: "Formatted at 2026-09-24 04:46:39 on dist-test-slave-hvcl"
server_key: "763a7159495bcfa4a81cb09883d66937"
server_key_iv: "b6ee8ff6392e319e93e47da465461297"
server_key_version: "encryptionkey@0"
I20260924 04:46:39.619803 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:46:39.622309 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:39.623700 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:46:39.623802 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "7da36394b3a144ceaeabb65a8ee94446"
format_stamp: "Formatted at 2026-09-24 04:46:39 on dist-test-slave-hvcl"
server_key: "763a7159495bcfa4a81cb09883d66937"
server_key_iv: "b6ee8ff6392e319e93e47da465461297"
server_key_version: "encryptionkey@0"
I20260924 04:46:39.623900 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:39.637452 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:39.640545 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:39.640722 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:39.641305 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:39.642113 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:39.642159 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:39.642211 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:39.642237 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:39.650516 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40039
I20260924 04:46:39.650717 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40039 every 8 connection(s)
I20260924 04:46:39.651472 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:46:39.656329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 1292
I20260924 04:46:39.656450 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:46:39.656703 392 external_mini_cluster.cc:1428] Setting key 5c105b736371e58e82369ab2a9fc431d
Sep 24 04:46:39 dist-test-slave-hvcl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:39.665020 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.653261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50865 (local address 127.0.98.62:45687)
0924 04:46:39.653487 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:39.653490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:39.654336 (+ 846us) server_negotiation.cc:407] Connection header received
0924 04:46:39.655711 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:39.655715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:39.655773 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:39.655853 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:39.657079 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.657766 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.659032 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.659266 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.659352 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.661771 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:39.661775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:39.661778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:39.661803 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:39.663109 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:39.663567 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:39.663571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:39.663572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:39.663654 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:39.663951 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:39.663955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:39.663957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:39.664143 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:39.664226 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:39.664715 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 04:46:39.664841 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":76,"threads_started":1}
I20260924 04:46:39.665874 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.653775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45687 (local address 127.0.98.3:50865)
0924 04:46:39.654198 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:39.654238 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:39.655500 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:39.656029 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:39.656036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:39.656379 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:39.656934 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:39.656949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.657877 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.657879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:39.658899 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:39.658908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.659398 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.659401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:39.659569 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.660140 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:39.660156 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:39.661598 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:39.663205 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:39.663210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:39.663227 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:39.663472 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:39.663733 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:39.663736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:39.663738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:39.663868 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:39.664236 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:39.664243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:39.664492 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 04:46:39.664662 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 04:46:39.664980 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":117,"threads_started":1}
I20260924 04:46:39.667012 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45687
I20260924 04:46:39.667260 1421 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:39.667745 1421 heartbeater.cc:511] Master 127.0.98.62:45687 requested a full tablet report, sending...
I20260924 04:46:39.668936 962 ts_manager.cc:194] Registered new tserver with Master: 7da36394b3a144ceaeabb65a8ee94446 (127.0.98.3:40039)
I20260924 04:46:39.669610 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50865
I20260924 04:46:39.670563 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:46:39.681927 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.673893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.0.98.62:45687)
0924 04:46:39.674043 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:39.674046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:39.674188 (+ 142us) server_negotiation.cc:407] Connection header received
0924 04:46:39.674298 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:39.674300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:39.674344 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:39.674497 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:39.675197 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.675970 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.677500 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.677724 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.677807 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.679086 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:39.679090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:39.679093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:39.679119 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:39.680389 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:39.680754 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:39.680757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:39.680759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:39.680834 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:39.681077 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:39.681080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:39.681082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:39.681299 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:39.681425 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:39.681644 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 04:46:39.681765 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 04:46:39.686827 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:46:39.689891 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:46:39.704537 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.698483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40039 (local address 127.0.0.1:40798)
0924 04:46:39.698866 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:39.698918 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:39.699090 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:39.700160 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:39.700166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:39.700190 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:39.700327 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:39.700334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.701758 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.701761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:39.703154 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:39.703162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.704098 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.704101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:39.704174 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.704204 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:46:39.704277 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:46:39.704378 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":108,"thread_start_us":129,"threads_started":1}
I20260924 04:46:39.704566 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.698817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40798 (local address 127.0.98.3:40039)
0924 04:46:39.699424 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:39.699436 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:39.699462 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:46:39.699540 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:39.699546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:39.699764 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:39.699923 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:39.700444 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.701632 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.703258 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.703995 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.704103 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.704256 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:39.704309 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 04:46:39.704401 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":156,"threads_started":1}
I20260924 04:46:39.704731 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.699268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36497 (local address 127.0.0.1:46672)
0924 04:46:39.700212 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:39.700224 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:39.700302 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:39.700694 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:39.700698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:39.700714 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:39.700788 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:39.700794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.701731 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.701735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:39.703361 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:39.703371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.704368 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.704371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:39.704454 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.704488 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 04:46:39.704553 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:46:39.704614 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":102,"thread_start_us":91,"threads_started":1}
I20260924 04:46:39.705157 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.699717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46672 (local address 127.0.98.1:36497)
0924 04:46:39.700040 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:39.700046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:39.700360 (+ 314us) server_negotiation.cc:407] Connection header received
0924 04:46:39.700439 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:39.700444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:39.700589 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:39.700706 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:39.700871 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.701619 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.703483 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.704259 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.704359 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.704479 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:39.704923 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 04:46:39.705023 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":62,"threads_started":1}
I20260924 04:46:39.706027 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.698524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38187 (local address 127.0.0.1:44236)
0924 04:46:39.699367 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:39.699380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:39.699463 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:39.700214 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:39.700217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:39.700235 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:39.700343 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:39.700349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.701785 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.701788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:39.703361 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:39.703371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.705644 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.705647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:39.705736 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.705760 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:46:39.705831 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:46:39.705901 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":200,"thread_start_us":76,"threads_started":1}
I20260924 04:46:39.706141 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.699324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44236 (local address 127.0.98.2:38187)
0924 04:46:39.699786 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:39.699793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:39.699815 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:46:39.699895 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:39.699901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:39.700092 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:39.700228 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:39.700460 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.701632 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.704542 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.705300 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.705400 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.705532 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:39.705866 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 04:46:39.706010 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":65,"threads_started":1}
I20260924 04:46:39.707823 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 83c8868ef042414995b6dee12a2ea86c (DEFAULT_TABLE table=test-table [id=eb428daa279342eda013ba2ca3e200f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:39.708983 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c8868ef042414995b6dee12a2ea86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:39.709149 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 83c8868ef042414995b6dee12a2ea86c (DEFAULT_TABLE table=test-table [id=eb428daa279342eda013ba2ca3e200f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:39.709615 1206 tablet_service.cc:1467] Processing CreateTablet for tablet 83c8868ef042414995b6dee12a2ea86c (DEFAULT_TABLE table=test-table [id=eb428daa279342eda013ba2ca3e200f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:39.710076 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c8868ef042414995b6dee12a2ea86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:39.710512 1206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c8868ef042414995b6dee12a2ea86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:39.716089 1440 tablet_bootstrap.cc:492] T 83c8868ef042414995b6dee12a2ea86c P 554d11e0726f4f6bad8c56d5cd0bb17c: Bootstrap starting.
I20260924 04:46:39.716990 1441 tablet_bootstrap.cc:492] T 83c8868ef042414995b6dee12a2ea86c P 7da36394b3a144ceaeabb65a8ee94446: Bootstrap starting.
I20260924 04:46:39.717589 1442 tablet_bootstrap.cc:492] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541: Bootstrap starting.
I20260924 04:46:39.718438 1440 tablet_bootstrap.cc:654] T 83c8868ef042414995b6dee12a2ea86c P 554d11e0726f4f6bad8c56d5cd0bb17c: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:39.718990 1441 tablet_bootstrap.cc:654] T 83c8868ef042414995b6dee12a2ea86c P 7da36394b3a144ceaeabb65a8ee94446: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:39.719352 1440 log.cc:824] T 83c8868ef042414995b6dee12a2ea86c P 554d11e0726f4f6bad8c56d5cd0bb17c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:39.720034 1442 tablet_bootstrap.cc:654] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:39.720172 1441 log.cc:824] T 83c8868ef042414995b6dee12a2ea86c P 7da36394b3a144ceaeabb65a8ee94446: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:39.720788 1442 log.cc:824] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:39.721880 1440 tablet_bootstrap.cc:492] T 83c8868ef042414995b6dee12a2ea86c P 554d11e0726f4f6bad8c56d5cd0bb17c: No bootstrap required, opened a new log
I20260924 04:46:39.722065 1440 ts_tablet_manager.cc:1397] T 83c8868ef042414995b6dee12a2ea86c P 554d11e0726f4f6bad8c56d5cd0bb17c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:46:39.722066 1441 tablet_bootstrap.cc:492] T 83c8868ef042414995b6dee12a2ea86c P 7da36394b3a144ceaeabb65a8ee94446: No bootstrap required, opened a new log
I20260924 04:46:39.722235 1441 ts_tablet_manager.cc:1397] T 83c8868ef042414995b6dee12a2ea86c P 7da36394b3a144ceaeabb65a8ee94446: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:46:39.722609 1442 tablet_bootstrap.cc:492] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541: No bootstrap required, opened a new log
I20260924 04:46:39.722783 1442 ts_tablet_manager.cc:1397] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:46:39.725581 1440 raft_consensus.cc:348] T 83c8868ef042414995b6dee12a2ea86c P 554d11e0726f4f6bad8c56d5cd0bb17c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da36394b3a144ceaeabb65a8ee94446" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "7f734556919a45038b38f8cca47d5541" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38187 } } peers { permanent_uuid: "554d11e0726f4f6bad8c56d5cd0bb17c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36497 } }
I20260924 04:46:39.725853 1440 raft_consensus.cc:374] T 83c8868ef042414995b6dee12a2ea86c P 554d11e0726f4f6bad8c56d5cd0bb17c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:39.725905 1440 raft_consensus.cc:729] T 83c8868ef042414995b6dee12a2ea86c P 554d11e0726f4f6bad8c56d5cd0bb17c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554d11e0726f4f6bad8c56d5cd0bb17c, State: Initialized, Role: FOLLOWER
I20260924 04:46:39.726116 1441 raft_consensus.cc:348] T 83c8868ef042414995b6dee12a2ea86c P 7da36394b3a144ceaeabb65a8ee94446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da36394b3a144ceaeabb65a8ee94446" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "7f734556919a45038b38f8cca47d5541" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38187 } } peers { permanent_uuid: "554d11e0726f4f6bad8c56d5cd0bb17c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36497 } }
I20260924 04:46:39.726418 1440 consensus_queue.cc:260] T 83c8868ef042414995b6dee12a2ea86c P 554d11e0726f4f6bad8c56d5cd0bb17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da36394b3a144ceaeabb65a8ee94446" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "7f734556919a45038b38f8cca47d5541" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38187 } } peers { permanent_uuid: "554d11e0726f4f6bad8c56d5cd0bb17c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36497 } }
I20260924 04:46:39.726507 1441 raft_consensus.cc:374] T 83c8868ef042414995b6dee12a2ea86c P 7da36394b3a144ceaeabb65a8ee94446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:39.726564 1441 raft_consensus.cc:729] T 83c8868ef042414995b6dee12a2ea86c P 7da36394b3a144ceaeabb65a8ee94446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da36394b3a144ceaeabb65a8ee94446, State: Initialized, Role: FOLLOWER
I20260924 04:46:39.727046 1441 consensus_queue.cc:260] T 83c8868ef042414995b6dee12a2ea86c P 7da36394b3a144ceaeabb65a8ee94446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da36394b3a144ceaeabb65a8ee94446" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "7f734556919a45038b38f8cca47d5541" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38187 } } peers { permanent_uuid: "554d11e0726f4f6bad8c56d5cd0bb17c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36497 } }
I20260924 04:46:39.727494 1440 ts_tablet_manager.cc:1428] T 83c8868ef042414995b6dee12a2ea86c P 554d11e0726f4f6bad8c56d5cd0bb17c: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260924 04:46:39.727423 1442 raft_consensus.cc:348] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da36394b3a144ceaeabb65a8ee94446" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "7f734556919a45038b38f8cca47d5541" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38187 } } peers { permanent_uuid: "554d11e0726f4f6bad8c56d5cd0bb17c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36497 } }
I20260924 04:46:39.727648 1442 raft_consensus.cc:374] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:39.727691 1442 raft_consensus.cc:729] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f734556919a45038b38f8cca47d5541, State: Initialized, Role: FOLLOWER
I20260924 04:46:39.727990 1441 ts_tablet_manager.cc:1428] T 83c8868ef042414995b6dee12a2ea86c P 7da36394b3a144ceaeabb65a8ee94446: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:46:39.728348 1421 heartbeater.cc:503] Master 127.0.98.62:45687 was elected leader, sending a full tablet report...
I20260924 04:46:39.728883 1442 consensus_queue.cc:260] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da36394b3a144ceaeabb65a8ee94446" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "7f734556919a45038b38f8cca47d5541" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38187 } } peers { permanent_uuid: "554d11e0726f4f6bad8c56d5cd0bb17c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36497 } }
I20260924 04:46:39.729796 1442 ts_tablet_manager.cc:1428] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260924 04:46:39.793501 1448 raft_consensus.cc:482] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:46:39.793766 1448 raft_consensus.cc:504] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da36394b3a144ceaeabb65a8ee94446" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "7f734556919a45038b38f8cca47d5541" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38187 } } peers { permanent_uuid: "554d11e0726f4f6bad8c56d5cd0bb17c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36497 } }
W20260924 04:46:39.794258 1288 tablet.cc:2369] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:46:39.795114 1448 leader_election.cc:288] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7da36394b3a144ceaeabb65a8ee94446 (127.0.98.3:40039), 554d11e0726f4f6bad8c56d5cd0bb17c (127.0.98.1:36497)
I20260924 04:46:39.799857 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.795522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55099 (local address 127.0.98.3:40039)
0924 04:46:39.795681 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:39.795685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:39.795927 (+ 242us) server_negotiation.cc:407] Connection header received
0924 04:46:39.796148 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:39.796152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:39.796199 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:39.796246 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:39.796667 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.797262 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.798464 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.799143 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.799247 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.799337 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:39.799638 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 04:46:39.799715 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 04:46:39.800143 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.795909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36497 (local address 127.0.98.2:60117)
0924 04:46:39.796190 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:39.796203 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:39.796305 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:39.796582 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:39.796584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:39.796602 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:39.796734 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:39.796741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.797616 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.797619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:39.798596 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:39.798605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.799715 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.799718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:39.799828 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.799850 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:46:39.799926 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 04:46:39.799995 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":41,"thread_start_us":90,"threads_started":1}
I20260924 04:46:39.800420 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.795484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40039 (local address 127.0.98.2:55099)
0924 04:46:39.795813 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:39.795827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:39.796014 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:39.796390 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:39.796402 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:39.796422 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:39.796548 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:39.796555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.797381 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.797384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:39.798345 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:39.798354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.799398 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.799401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:39.799498 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.799522 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:46:39.800023 (+ 501us) client_negotiation.cc:240] Negotiation successful
0924 04:46:39.800101 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":31,"thread_start_us":138,"threads_started":1}
I20260924 04:46:39.800427 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.796080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60117 (local address 127.0.98.1:36497)
0924 04:46:39.796212 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:39.796216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:39.796271 (+ 55us) server_negotiation.cc:407] Connection header received
0924 04:46:39.796411 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:39.796414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:39.796462 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:39.796534 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:39.796889 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.797476 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.798746 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.799382 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.799495 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.799647 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:39.800217 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 04:46:39.800295 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:46:39.800980 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c8868ef042414995b6dee12a2ea86c" candidate_uuid: "7f734556919a45038b38f8cca47d5541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554d11e0726f4f6bad8c56d5cd0bb17c" is_pre_election: true
I20260924 04:46:39.800992 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c8868ef042414995b6dee12a2ea86c" candidate_uuid: "7f734556919a45038b38f8cca47d5541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da36394b3a144ceaeabb65a8ee94446" is_pre_election: true
I20260924 04:46:39.801287 1107 raft_consensus.cc:2454] T 83c8868ef042414995b6dee12a2ea86c P 554d11e0726f4f6bad8c56d5cd0bb17c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f734556919a45038b38f8cca47d5541 in term 0.
I20260924 04:46:39.801290 1373 raft_consensus.cc:2454] T 83c8868ef042414995b6dee12a2ea86c P 7da36394b3a144ceaeabb65a8ee94446 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f734556919a45038b38f8cca47d5541 in term 0.
I20260924 04:46:39.801733 1177 leader_election.cc:302] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da36394b3a144ceaeabb65a8ee94446, 7f734556919a45038b38f8cca47d5541; no voters:
I20260924 04:46:39.801990 1448 raft_consensus.cc:2781] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:46:39.802095 1448 raft_consensus.cc:482] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:46:39.802145 1448 raft_consensus.cc:3037] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [term 0 FOLLOWER]: Advancing to term 1
W20260924 04:46:39.825727 1154 tablet.cc:2369] T 83c8868ef042414995b6dee12a2ea86c P 554d11e0726f4f6bad8c56d5cd0bb17c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:46:39.846204 1448 raft_consensus.cc:504] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da36394b3a144ceaeabb65a8ee94446" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "7f734556919a45038b38f8cca47d5541" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38187 } } peers { permanent_uuid: "554d11e0726f4f6bad8c56d5cd0bb17c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36497 } }
I20260924 04:46:39.846690 1448 leader_election.cc:288] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [CANDIDATE]: Term 1 election: Requested vote from peers 7da36394b3a144ceaeabb65a8ee94446 (127.0.98.3:40039), 554d11e0726f4f6bad8c56d5cd0bb17c (127.0.98.1:36497)
I20260924 04:46:39.847234 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c8868ef042414995b6dee12a2ea86c" candidate_uuid: "7f734556919a45038b38f8cca47d5541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da36394b3a144ceaeabb65a8ee94446"
I20260924 04:46:39.847278 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c8868ef042414995b6dee12a2ea86c" candidate_uuid: "7f734556919a45038b38f8cca47d5541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "554d11e0726f4f6bad8c56d5cd0bb17c"
I20260924 04:46:39.847379 1373 raft_consensus.cc:3037] T 83c8868ef042414995b6dee12a2ea86c P 7da36394b3a144ceaeabb65a8ee94446 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:39.847445 1107 raft_consensus.cc:3037] T 83c8868ef042414995b6dee12a2ea86c P 554d11e0726f4f6bad8c56d5cd0bb17c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:39.848517 1373 raft_consensus.cc:2454] T 83c8868ef042414995b6dee12a2ea86c P 7da36394b3a144ceaeabb65a8ee94446 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f734556919a45038b38f8cca47d5541 in term 1.
I20260924 04:46:39.848740 1107 raft_consensus.cc:2454] T 83c8868ef042414995b6dee12a2ea86c P 554d11e0726f4f6bad8c56d5cd0bb17c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f734556919a45038b38f8cca47d5541 in term 1.
I20260924 04:46:39.848905 1177 leader_election.cc:302] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da36394b3a144ceaeabb65a8ee94446, 7f734556919a45038b38f8cca47d5541; no voters:
I20260924 04:46:39.849102 1448 raft_consensus.cc:2781] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:39.849318 1448 raft_consensus.cc:686] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [term 1 LEADER]: Becoming Leader. State: Replica: 7f734556919a45038b38f8cca47d5541, State: Running, Role: LEADER
I20260924 04:46:39.849725 1448 consensus_queue.cc:237] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da36394b3a144ceaeabb65a8ee94446" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } } peers { permanent_uuid: "7f734556919a45038b38f8cca47d5541" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38187 } } peers { permanent_uuid: "554d11e0726f4f6bad8c56d5cd0bb17c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36497 } }
I20260924 04:46:39.852945 962 catalog_manager.cc:5495] T 83c8868ef042414995b6dee12a2ea86c P 7f734556919a45038b38f8cca47d5541 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f734556919a45038b38f8cca47d5541 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7f734556919a45038b38f8cca47d5541" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7da36394b3a144ceaeabb65a8ee94446" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f734556919a45038b38f8cca47d5541" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "554d11e0726f4f6bad8c56d5cd0bb17c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36497 } health_report { overall_health: UNKNOWN } } }
Sep 24 04:46:39 dist-test-slave-hvcl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
W20260924 04:46:39.902462 1422 tablet.cc:2369] T 83c8868ef042414995b6dee12a2ea86c P 7da36394b3a144ceaeabb65a8ee94446: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:46:39.906365 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.892507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.0.98.1:36497)
0924 04:46:39.892657 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:39.892663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:39.892852 (+ 189us) server_negotiation.cc:407] Connection header received
0924 04:46:39.892976 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:39.892979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:39.893034 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:39.893133 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:39.893827 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.894389 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.895261 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.895478 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.895565 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.903065 (+ 7500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:39.903077 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:39.903091 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:39.903129 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:39.904624 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:39.905060 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:39.905066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:39.905067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:39.905156 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:39.905410 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:39.905413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:39.905415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:39.905670 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:39.905883 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:39.906061 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 04:46:39.906186 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 04:46:39 dist-test-slave-hvcl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225199, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:46:39 dist-test-slave-hvcl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225199, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:46:39.954691 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:39.945730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46692 (local address 127.0.98.1:36497)
0924 04:46:39.945883 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:39.945888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:39.945907 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:46:39.946042 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:39.946045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:39.946094 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:39.946176 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:39.946845 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.947640 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.948496 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:39.948863 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:39.948961 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:39.951481 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:39.951484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:39.951486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:39.951513 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:39.952848 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:39.953293 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:39.953297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:39.953298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:39.953380 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:39.953725 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:39.953728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:39.953730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:39.953937 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:39.954057 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:39.954347 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 04:46:39.954526 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 04:46:39 dist-test-slave-hvcl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:46:40 dist-test-slave-hvcl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:46:40.055301 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:40.044054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46696 (local address 127.0.98.1:36497)
0924 04:46:40.044238 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:40.044243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:40.044340 (+ 97us) server_negotiation.cc:407] Connection header received
0924 04:46:40.044465 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:40.044469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:40.044520 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:40.044613 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:40.045603 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:40.046460 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:40.047449 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:40.047694 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:40.047792 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:40.051421 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:40.051425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:40.051427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:40.051459 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:40.053219 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:40.053815 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:40.053819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:40.053820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:40.053908 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:40.054283 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:40.054286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:40.054287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:40.054478 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:40.054690 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:40.055034 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 04:46:40.055165 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
W20260924 04:46:40.056715 1067 tablet_service.cc:2908] Rejecting scan request for tablet 83c8868ef042414995b6dee12a2ea86c: Uninitialized: clean time has not yet been initialized
I20260924 04:46:40.059355 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 1024
I20260924 04:46:40.064986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 1158
I20260924 04:46:40.070034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 1292
I20260924 04:46:40.075024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 930
2026-09-24T04:46:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3301 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:46:41 dist-test-slave-hvcl krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:46:41 dist-test-slave-hvcl krb5kdc[1465](info): set up 2 sockets
Sep 24 04:46:41 dist-test-slave-hvcl krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260924 04:46:42.244504 392 mini_kdc.cc:121] Time spent starting KDC: real 2.148s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 04:46:42.385095 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.140s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 04:46:42.505998 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.121s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 04:46:42.666811 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.161s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/test-user.keytab.
W20260924 04:46:43.236656 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.570s user 0.001s sys 0.000s
Sep 24 04:46:43 dist-test-slave-hvcl krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:46:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:46:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260924 04:46:44.123252 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.865s user 0.000s sys 0.003s
I20260924 04:46:44.123744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41415
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:44.235620 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:44.235899 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:44.235939 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:44.239921 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:46:44.239984 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:46:44.240000 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:44.240015 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:44.240031 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:46:44.240046 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:46:44.245414 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41415
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:44.246482 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:44.247504 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:44.256369 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:44.256378 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:44.256583 1478 server_base.cc:1034] running on GCE node
W20260924 04:46:44.256378 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:44.257254 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:44.258333 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:44.259483 1478 hybrid_clock.cc:648] HybridClock initialized: now 1790225204259456 us; error 40 us; skew 500 ppm
Sep 24 04:46:44 dist-test-slave-hvcl krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:44.262462 1478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:46:44.263521 1478 webserver.cc:460] Webserver started at http://127.0.98.62:46137/ using document root <none> and password file <none>
I20260924 04:46:44.264088 1478 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:44.264151 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:44.264355 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:44.266415 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "a4104fd6d5574852b0b91c9bfdc7224d"
format_stamp: "Formatted at 2026-09-24 04:46:44 on dist-test-slave-hvcl"
server_key: "a41ba0f6ae0dace220c83676c4db03f8"
server_key_iv: "9b34cb47db67432eddee58cbd5dcb1fa"
server_key_version: "encryptionkey@0"
I20260924 04:46:44.267058 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4104fd6d5574852b0b91c9bfdc7224d"
format_stamp: "Formatted at 2026-09-24 04:46:44 on dist-test-slave-hvcl"
server_key: "a41ba0f6ae0dace220c83676c4db03f8"
server_key_iv: "9b34cb47db67432eddee58cbd5dcb1fa"
server_key_version: "encryptionkey@0"
I20260924 04:46:44.270881 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:46:44.273428 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:44.274818 1478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:46:44.274951 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "a4104fd6d5574852b0b91c9bfdc7224d"
format_stamp: "Formatted at 2026-09-24 04:46:44 on dist-test-slave-hvcl"
server_key: "a41ba0f6ae0dace220c83676c4db03f8"
server_key_iv: "9b34cb47db67432eddee58cbd5dcb1fa"
server_key_version: "encryptionkey@0"
I20260924 04:46:44.275092 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:44.295683 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:44.299108 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:44.299295 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:44.306939 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41415
I20260924 04:46:44.306959 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41415 every 8 connection(s)
I20260924 04:46:44.307945 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:46:44.309916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 1478
I20260924 04:46:44.310011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:46:44.310253 392 external_mini_cluster.cc:1428] Setting key 8e318adc842786c80ae21c5ceef129d2
I20260924 04:46:44.311022 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:44.317713 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d: Bootstrap starting.
Sep 24 04:46:44 dist-test-slave-hvcl krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:44.320115 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:44.321100 1556 log.cc:824] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:44.322954 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d: No bootstrap required, opened a new log
I20260924 04:46:44.324388 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:44.311622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.0.98.62:41415)
0924 04:46:44.312029 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:44.312048 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:44.312090 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:46:44.312934 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:44.312958 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:44.313371 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:44.313751 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:44.314351 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.315850 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.316711 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.317363 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.317513 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:44.319975 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:44.319980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:44.319994 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:44.320021 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:44.321705 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:44.322128 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:44.322133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:44.322137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:44.322247 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:44.322506 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:44.322509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:44.322510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:44.322894 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:44.323106 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:44.323363 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 04:46:44.323659 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":116,"threads_started":1}
I20260924 04:46:44.326304 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4104fd6d5574852b0b91c9bfdc7224d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } }
I20260924 04:46:44.326570 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:44.326622 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4104fd6d5574852b0b91c9bfdc7224d, State: Initialized, Role: FOLLOWER
I20260924 04:46:44.327085 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4104fd6d5574852b0b91c9bfdc7224d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } }
I20260924 04:46:44.327217 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:46:44.327281 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:46:44.327385 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:44.328306 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4104fd6d5574852b0b91c9bfdc7224d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } }
I20260924 04:46:44.328656 1556 leader_election.cc:302] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4104fd6d5574852b0b91c9bfdc7224d; no voters:
I20260924 04:46:44.328954 1556 leader_election.cc:288] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:46:44.329102 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:44.329321 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [term 1 LEADER]: Becoming Leader. State: Replica: a4104fd6d5574852b0b91c9bfdc7224d, State: Running, Role: LEADER
I20260924 04:46:44.329676 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4104fd6d5574852b0b91c9bfdc7224d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } }
I20260924 04:46:44.330149 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:46:44.331313 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4104fd6d5574852b0b91c9bfdc7224d. Latest consensus state: current_term: 1 leader_uuid: "a4104fd6d5574852b0b91c9bfdc7224d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4104fd6d5574852b0b91c9bfdc7224d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } }
I20260924 04:46:44.331439 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:44.331821 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:46:44.331785 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4104fd6d5574852b0b91c9bfdc7224d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4104fd6d5574852b0b91c9bfdc7224d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41415 } } }
I20260924 04:46:44.332675 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:44.334794 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:46:44.339789 1570 catalog_manager.cc:1347] Generated new cluster ID: e8dff0496d7c4b5e862b160a69a9732c
I20260924 04:46:44.339859 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:46:44.351302 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:46:44.352485 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:46:44.362497 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d: Generated new TSK 0
I20260924 04:46:44.363454 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:46:44.394536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41415
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:44.506446 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:44.506728 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:44.506789 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:44.510843 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:44.510910 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:44.510984 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:46:44.511040 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:46:44.516476 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:44.517650 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:44.518676 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:44.528398 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:44.528465 1582 server_base.cc:1034] running on GCE node
W20260924 04:46:44.528394 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:44.528394 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:44.529071 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:44.529696 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:44.530860 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790225204530830 us; error 41 us; skew 500 ppm
Sep 24 04:46:44 dist-test-slave-hvcl krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:44.533638 1582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:46:44.534618 1582 webserver.cc:460] Webserver started at http://127.0.98.1:37329/ using document root <none> and password file <none>
I20260924 04:46:44.535197 1582 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:44.535256 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:44.535422 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:44.537492 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "963e47baa4e04d22ad511211727d9977"
format_stamp: "Formatted at 2026-09-24 04:46:44 on dist-test-slave-hvcl"
server_key: "4e0310a01db8ae280c2e43ff705e0219"
server_key_iv: "73c59b1948bbcd550213c3f08d156a9f"
server_key_version: "encryptionkey@0"
I20260924 04:46:44.538038 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "963e47baa4e04d22ad511211727d9977"
format_stamp: "Formatted at 2026-09-24 04:46:44 on dist-test-slave-hvcl"
server_key: "4e0310a01db8ae280c2e43ff705e0219"
server_key_iv: "73c59b1948bbcd550213c3f08d156a9f"
server_key_version: "encryptionkey@0"
I20260924 04:46:44.541853 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:46:44.544466 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:44.545840 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:46:44.545976 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "963e47baa4e04d22ad511211727d9977"
format_stamp: "Formatted at 2026-09-24 04:46:44 on dist-test-slave-hvcl"
server_key: "4e0310a01db8ae280c2e43ff705e0219"
server_key_iv: "73c59b1948bbcd550213c3f08d156a9f"
server_key_version: "encryptionkey@0"
I20260924 04:46:44.546087 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:44.567255 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:44.570344 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:44.570520 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:44.571939 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:44.571987 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:44.572039 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:44.572064 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:46:44 dist-test-slave-hvcl krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:44.585162 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39869
I20260924 04:46:44.585397 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39869 every 8 connection(s)
I20260924 04:46:44.586205 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:46:44.588421 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:44.572702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34347 (local address 127.0.98.62:41415)
0924 04:46:44.572860 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:44.572866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:44.573714 (+ 848us) server_negotiation.cc:407] Connection header received
0924 04:46:44.574802 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:44.574806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:44.574862 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:44.574940 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:44.576310 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.577105 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.581389 (+ 4284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.581743 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.581875 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:44.584940 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:44.584944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:44.584947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:44.584976 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:44.586341 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:44.587071 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:44.587074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:44.587076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:44.587149 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:44.587420 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:44.587422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:44.587423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:44.587579 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:44.587647 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:44.588186 (+ 539us) server_negotiation.cc:299] Negotiation successful
0924 04:46:44.588302 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:46:44.589125 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:44.573079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41415 (local address 127.0.98.1:34347)
0924 04:46:44.573543 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:44.573593 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:44.574540 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:44.575185 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:44.575192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:44.575534 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:44.576156 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:44.576178 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.577240 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.577244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:44.581237 (+ 3993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:44.581252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.581942 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.581947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:44.582177 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:44.582806 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:44.582828 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:44.584743 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:44.586443 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:44.586450 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:44.586467 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:44.586955 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:44.587229 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:44.587233 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:44.587235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:44.587361 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:44.587662 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:44.587668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:44.587912 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 04:46:44.588061 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 04:46:44.588344 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260924 04:46:44.590356 1713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41415
I20260924 04:46:44.590490 1713 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:44.591140 1713 heartbeater.cc:511] Master 127.0.98.62:41415 requested a full tablet report, sending...
I20260924 04:46:44.591969 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 1582
I20260924 04:46:44.592077 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:46:44.592352 392 external_mini_cluster.cc:1428] Setting key 64293a8a37928402260469d55a742833
I20260924 04:46:44.592994 1506 ts_manager.cc:194] Registered new tserver with Master: 963e47baa4e04d22ad511211727d9977 (127.0.98.1:39869)
I20260924 04:46:44.594355 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34347
I20260924 04:46:44.599792 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:44.592736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41415 (local address 127.0.98.1:46507)
0924 04:46:44.593071 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:44.593085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:44.593203 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:44.593524 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:44.593526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:44.593734 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:44.593849 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:44.593856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.594796 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.594799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:44.595573 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:44.595581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.596000 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.596003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:44.596089 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:44.596559 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:44.596574 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:44.596889 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:44.598425 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:44.598429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:44.598431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:44.598609 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:44.598843 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:44.598846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:44.598848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:44.598925 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:44.599283 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:44.599286 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:44.599372 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 04:46:44.599454 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:46:44.599594 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":88,"threads_started":1}
I20260924 04:46:44.599787 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:44.592720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46507 (local address 127.0.98.62:41415)
0924 04:46:44.592827 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:44.592832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:44.593145 (+ 313us) server_negotiation.cc:407] Connection header received
0924 04:46:44.593300 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:44.593303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:44.593347 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:44.593421 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:44.593970 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.594668 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.595678 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.595902 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.595986 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:44.597011 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:44.597014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:44.597017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:44.597040 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:44.598327 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:44.598691 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:44.598694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:44.598696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:44.598766 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:44.599022 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:44.599026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:44.599027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:44.599198 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:44.599282 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:44.599515 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 04:46:44.599615 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:46:44.622637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41415
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:44.735116 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:44.735379 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:44.735423 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:44.739461 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:44.739516 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:44.739554 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:46:44.739593 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:46:44.744766 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:44.745779 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:44.746758 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:44.756443 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:44.756438 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:44.756650 1722 server_base.cc:1034] running on GCE node
W20260924 04:46:44.756487 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:44.757171 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:44.757797 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:44.758958 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790225204758930 us; error 39 us; skew 500 ppm
Sep 24 04:46:44 dist-test-slave-hvcl krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:44.761698 1722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:46:44.762573 1722 webserver.cc:460] Webserver started at http://127.0.98.2:35591/ using document root <none> and password file <none>
I20260924 04:46:44.763131 1722 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:44.763191 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:44.763389 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:44.765409 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "3bc064ce6f074c3fa2bcc5c92d92a938"
format_stamp: "Formatted at 2026-09-24 04:46:44 on dist-test-slave-hvcl"
server_key: "d3e85fe9ce4b4097776e1619195802c3"
server_key_iv: "a2798e3a06ac04c7774cb133485cead6"
server_key_version: "encryptionkey@0"
I20260924 04:46:44.765942 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3bc064ce6f074c3fa2bcc5c92d92a938"
format_stamp: "Formatted at 2026-09-24 04:46:44 on dist-test-slave-hvcl"
server_key: "d3e85fe9ce4b4097776e1619195802c3"
server_key_iv: "a2798e3a06ac04c7774cb133485cead6"
server_key_version: "encryptionkey@0"
I20260924 04:46:44.769591 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260924 04:46:44.772221 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:44.773617 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:46:44.773717 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "3bc064ce6f074c3fa2bcc5c92d92a938"
format_stamp: "Formatted at 2026-09-24 04:46:44 on dist-test-slave-hvcl"
server_key: "d3e85fe9ce4b4097776e1619195802c3"
server_key_iv: "a2798e3a06ac04c7774cb133485cead6"
server_key_version: "encryptionkey@0"
I20260924 04:46:44.773824 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:44.786002 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:44.789171 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:44.789348 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:44.790856 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:44.790913 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:44.791000 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:44.791034 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:46:44 dist-test-slave-hvcl krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:44.800585 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33481
I20260924 04:46:44.800665 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33481 every 8 connection(s)
I20260924 04:46:44.801649 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:46:44.804308 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:44.791665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56249 (local address 127.0.98.62:41415)
0924 04:46:44.791807 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:44.791812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:44.792709 (+ 897us) server_negotiation.cc:407] Connection header received
0924 04:46:44.793764 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:44.793768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:44.793827 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:44.793885 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:44.795490 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.796112 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.797627 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.797848 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.797937 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:44.800567 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:44.800571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:44.800574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:44.800606 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:44.802105 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:44.802666 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:44.802669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:44.802670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:44.802759 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:44.803262 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:44.803264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:44.803266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:44.803463 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:44.803540 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:44.804068 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 04:46:44.804188 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:46:44.805107 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:44.792070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41415 (local address 127.0.98.2:56249)
0924 04:46:44.792537 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:44.792586 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:44.793507 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:44.794297 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:44.794307 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:44.794718 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:44.795333 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:44.795359 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.796241 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.796244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:44.797508 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:44.797520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.797988 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.797993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:44.798170 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:44.798822 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:44.798844 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:44.800360 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:44.802229 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:44.802260 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:44.802275 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:44.802566 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:44.802914 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:44.802917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:44.802918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:44.803168 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:44.803596 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:44.803601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:44.803857 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 04:46:44.804015 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 04:46:44.804287 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260924 04:46:44.806280 1853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41415
I20260924 04:46:44.806417 1853 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:44.806900 1853 heartbeater.cc:511] Master 127.0.98.62:41415 requested a full tablet report, sending...
I20260924 04:46:44.807977 1510 ts_manager.cc:194] Registered new tserver with Master: 3bc064ce6f074c3fa2bcc5c92d92a938 (127.0.98.2:33481)
I20260924 04:46:44.808693 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56249
I20260924 04:46:44.808799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 1722
I20260924 04:46:44.808902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:46:44.809187 392 external_mini_cluster.cc:1428] Setting key f9c275c3e4616abd5d443c33337228e9
I20260924 04:46:44.815383 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:44.808431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36753 (local address 127.0.98.62:41415)
0924 04:46:44.808542 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:44.808545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:44.808673 (+ 128us) server_negotiation.cc:407] Connection header received
0924 04:46:44.808810 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:44.808813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:44.808860 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:44.808937 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:44.809532 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.810253 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.811235 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.811445 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.811527 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:44.812642 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:44.812646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:44.812649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:44.812673 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:44.813885 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:44.814251 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:44.814255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:44.814257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:44.814325 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:44.814564 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:44.814567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:44.814569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:44.814773 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:44.814864 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:44.815124 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 04:46:44.815227 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:46:44.815510 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:44.808317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41415 (local address 127.0.98.2:36753)
0924 04:46:44.808586 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:44.808600 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:44.808696 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:44.809046 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:44.809049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:44.809260 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:44.809413 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:44.809421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.810409 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.810412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:44.811133 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:44.811142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:44.811538 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:44.811541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:44.811626 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:44.812132 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:44.812142 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:44.812490 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:44.813982 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:44.813985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:44.813987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:44.814168 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:44.814403 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:44.814406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:44.814413 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:44.814488 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:44.814862 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:44.814865 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:44.814955 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 04:46:44.815074 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 04:46:44.815218 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:46:44.837738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41415
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:44.950672 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:44.951076 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:44.951151 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:44.955510 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:44.955582 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:44.955634 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:46:44.955677 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:46:44.961373 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:44.962435 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:44.963507 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:44.973047 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:44.973188 1862 server_base.cc:1034] running on GCE node
W20260924 04:46:44.973040 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:44.973042 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:44.973734 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:44.974385 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:44.975569 1862 hybrid_clock.cc:648] HybridClock initialized: now 1790225204975547 us; error 37 us; skew 500 ppm
Sep 24 04:46:44 dist-test-slave-hvcl krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:44.982468 1862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:46:44.983520 1862 webserver.cc:460] Webserver started at http://127.0.98.3:36157/ using document root <none> and password file <none>
I20260924 04:46:44.984103 1862 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:44.984171 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:44.984354 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:44.986405 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "65225e65303c4788b4c52ebb6c1b4db4"
format_stamp: "Formatted at 2026-09-24 04:46:44 on dist-test-slave-hvcl"
server_key: "05d48c77304001806074391a8ff6f9c8"
server_key_iv: "1e6ed8ebe9df85ffdff33fa98dbab362"
server_key_version: "encryptionkey@0"
I20260924 04:46:44.986948 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "65225e65303c4788b4c52ebb6c1b4db4"
format_stamp: "Formatted at 2026-09-24 04:46:44 on dist-test-slave-hvcl"
server_key: "05d48c77304001806074391a8ff6f9c8"
server_key_iv: "1e6ed8ebe9df85ffdff33fa98dbab362"
server_key_version: "encryptionkey@0"
I20260924 04:46:44.990597 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:46:44.993129 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:44.994434 1862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:46:44.994544 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "65225e65303c4788b4c52ebb6c1b4db4"
format_stamp: "Formatted at 2026-09-24 04:46:44 on dist-test-slave-hvcl"
server_key: "05d48c77304001806074391a8ff6f9c8"
server_key_iv: "1e6ed8ebe9df85ffdff33fa98dbab362"
server_key_version: "encryptionkey@0"
I20260924 04:46:44.994644 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:45.010478 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:45.013660 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:45.013844 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:45.015318 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:45.015369 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:45.015422 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:45.015447 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:46:45 dist-test-slave-hvcl krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:45.025552 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37711
I20260924 04:46:45.025967 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37711 every 8 connection(s)
I20260924 04:46:45.026495 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:46:45.028630 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.016075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33729 (local address 127.0.98.62:41415)
0924 04:46:45.016191 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.016194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.017235 (+ 1041us) server_negotiation.cc:407] Connection header received
0924 04:46:45.018302 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.018306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.018365 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.018419 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:45.020026 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.020611 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.021724 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.021941 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.022030 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.024998 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:45.025001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:45.025003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:45.025030 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:45.026504 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:45.027167 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:45.027171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:45.027173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:45.027265 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:45.027592 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:45.027596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:45.027597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:45.027783 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:45.027866 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.028351 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.028470 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:46:45.029383 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.016615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41415 (local address 127.0.98.3:33729)
0924 04:46:45.017081 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:45.017125 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:45.018052 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:45.018848 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:45.018857 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:45.019249 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:45.019855 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.019873 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.020733 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.020736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.021606 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.021615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.022097 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.022101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.022277 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.022877 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:45.022896 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:45.024823 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:45.026623 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:45.026628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:45.026644 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:45.027039 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:45.027362 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:45.027365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:45.027367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:45.027497 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:45.027879 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:45.027886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:45.028117 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 04:46:45.028272 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 04:46:45.028581 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":131,"threads_started":1}
I20260924 04:46:45.030596 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41415
I20260924 04:46:45.030725 1993 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:45.031250 1993 heartbeater.cc:511] Master 127.0.98.62:41415 requested a full tablet report, sending...
I20260924 04:46:45.032202 1506 ts_manager.cc:194] Registered new tserver with Master: 65225e65303c4788b4c52ebb6c1b4db4 (127.0.98.3:37711)
I20260924 04:46:45.032935 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33729
I20260924 04:46:45.034159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 1862
I20260924 04:46:45.034304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:46:45.034584 392 external_mini_cluster.cc:1428] Setting key 2ffea65d1a6a2baa4a5e1330a5dcd3e2
I20260924 04:46:45.036504 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:46:45.039383 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.032659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47007 (local address 127.0.98.62:41415)
0924 04:46:45.032760 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.032764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.032929 (+ 165us) server_negotiation.cc:407] Connection header received
0924 04:46:45.033048 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.033051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.033095 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.033165 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:45.033619 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.034116 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.035054 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.035310 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.035409 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.036363 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:45.036366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:45.036369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:45.036392 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:45.037792 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:45.038253 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:45.038257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:45.038259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:45.038352 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:45.038579 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:45.038582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:45.038584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:45.038788 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:45.038877 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.039130 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.039228 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:46:45.039391 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.032583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41415 (local address 127.0.98.3:47007)
0924 04:46:45.032869 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:45.032881 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:45.032971 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:45.033193 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:45.033195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:45.033402 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:45.033519 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.033525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.034225 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.034228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.034915 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.034925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.035418 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.035420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.035490 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.035933 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:45.035945 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:45.036244 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:45.037893 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:45.037896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:45.037898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:45.038127 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:45.038434 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:45.038436 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:45.038438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:45.038507 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:45.038875 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:45.038877 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:45.038952 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 04:46:45.039072 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 04:46:45.039209 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
Sep 24 04:46:45 dist-test-slave-hvcl krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225205, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:46:45 dist-test-slave-hvcl krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225205, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:45.061681 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.053774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39328 (local address 127.0.98.62:41415)
0924 04:46:45.053894 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.053898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.053911 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:46:45.053955 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.053958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.053996 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.054072 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:45.054776 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.055345 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.056153 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.056378 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.056455 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.058843 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:45.058846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:45.058848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:45.058870 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:45.060191 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:45.060582 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:45.060585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:45.060586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:45.060664 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:45.060928 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:45.060931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:45.060932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:45.061107 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:45.061219 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.061462 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.061554 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:46:45.064447 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:46:45.067449 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:46:45.081933 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.075840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39869 (local address 127.0.0.1:59304)
0924 04:46:45.076306 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:45.076354 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:45.076486 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:45.077260 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:45.077266 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:45.077287 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:45.077453 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.077461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.079033 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.079037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.080550 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.080555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.081401 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.081405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.081516 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.081555 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 04:46:45.081658 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 04:46:45.081776 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":155,"thread_start_us":90,"threads_started":1}
I20260924 04:46:45.081939 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.076013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.0.98.1:39869)
0924 04:46:45.076762 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.076776 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.076799 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:46:45.076872 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.076877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.077089 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.077597 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:45.077737 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.078903 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.080666 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.081295 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.081389 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.081568 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.081686 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.081776 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":89,"threads_started":1}
I20260924 04:46:45.082366 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.076593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33481 (local address 127.0.0.1:43798)
0924 04:46:45.077447 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:45.077460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:45.077597 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:45.078275 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:45.078279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:45.078298 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:45.078408 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.078413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.079806 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.079809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.080770 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.080779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.081935 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.081938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.082106 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.082127 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:46:45.082196 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:46:45.082260 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":3,"thread_start_us":175,"threads_started":1}
I20260924 04:46:45.082476 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.076967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.0.98.2:33481)
0924 04:46:45.077308 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.077318 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.077534 (+ 216us) server_negotiation.cc:407] Connection header received
0924 04:46:45.077735 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.077741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.077925 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.078036 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:45.078535 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.079696 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.080887 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.081633 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.081724 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.081883 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.082241 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.082332 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":83,"threads_started":1}
I20260924 04:46:45.082532 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.076817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.0.98.3:37711)
0924 04:46:45.077218 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.077229 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.077251 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:46:45.077321 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.077326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.077520 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.077669 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:45.077860 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.078843 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.080696 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.081467 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.081568 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.081723 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.082284 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.082387 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":101,"threads_started":1}
I20260924 04:46:45.082191 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.075996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37711 (local address 127.0.0.1:34940)
0924 04:46:45.076707 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:45.076718 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:45.076797 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:45.077648 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:45.077651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:45.077666 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:45.077753 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.077759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.078943 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.078946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.080544 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.080554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.081785 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.081789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.081884 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.081907 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:46:45.081973 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:46:45.082044 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":47,"thread_start_us":107,"threads_started":1}
I20260924 04:46:45.085216 1640 tablet_service.cc:1467] Processing CreateTablet for tablet e443299b09ba4cfc95868cdb5b93b48a (DEFAULT_TABLE table=test-table [id=43bbd9ac68cd4917933e431620cef5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:45.085273 1787 tablet_service.cc:1467] Processing CreateTablet for tablet e443299b09ba4cfc95868cdb5b93b48a (DEFAULT_TABLE table=test-table [id=43bbd9ac68cd4917933e431620cef5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:45.086176 1640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e443299b09ba4cfc95868cdb5b93b48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:45.086247 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e443299b09ba4cfc95868cdb5b93b48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:45.085934 1927 tablet_service.cc:1467] Processing CreateTablet for tablet e443299b09ba4cfc95868cdb5b93b48a (DEFAULT_TABLE table=test-table [id=43bbd9ac68cd4917933e431620cef5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:45.086753 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e443299b09ba4cfc95868cdb5b93b48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:45.093184 2016 tablet_bootstrap.cc:492] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977: Bootstrap starting.
I20260924 04:46:45.094005 2017 tablet_bootstrap.cc:492] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4: Bootstrap starting.
I20260924 04:46:45.094338 2018 tablet_bootstrap.cc:492] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938: Bootstrap starting.
I20260924 04:46:45.095263 2016 tablet_bootstrap.cc:654] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:45.095966 2016 log.cc:824] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:45.096269 2018 tablet_bootstrap.cc:654] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:45.096910 2017 tablet_bootstrap.cc:654] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:45.097080 2018 log.cc:824] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:45.097669 2017 log.cc:824] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:45.097882 2016 tablet_bootstrap.cc:492] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977: No bootstrap required, opened a new log
I20260924 04:46:45.098078 2016 ts_tablet_manager.cc:1397] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:46:45.100703 2017 tablet_bootstrap.cc:492] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4: No bootstrap required, opened a new log
I20260924 04:46:45.100703 2018 tablet_bootstrap.cc:492] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938: No bootstrap required, opened a new log
I20260924 04:46:45.100888 2018 ts_tablet_manager.cc:1397] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260924 04:46:45.100888 2017 ts_tablet_manager.cc:1397] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260924 04:46:45.101217 2016 raft_consensus.cc:348] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } }
I20260924 04:46:45.101450 2016 raft_consensus.cc:374] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:45.101526 2016 raft_consensus.cc:729] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 963e47baa4e04d22ad511211727d9977, State: Initialized, Role: FOLLOWER
I20260924 04:46:45.101944 2016 consensus_queue.cc:260] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } }
I20260924 04:46:45.102548 1713 heartbeater.cc:503] Master 127.0.98.62:41415 was elected leader, sending a full tablet report...
I20260924 04:46:45.102866 2016 ts_tablet_manager.cc:1428] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:46:45.104672 2018 raft_consensus.cc:348] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } }
I20260924 04:46:45.104934 2017 raft_consensus.cc:348] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } }
I20260924 04:46:45.105100 2018 raft_consensus.cc:374] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:45.105158 2018 raft_consensus.cc:729] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc064ce6f074c3fa2bcc5c92d92a938, State: Initialized, Role: FOLLOWER
I20260924 04:46:45.105203 2017 raft_consensus.cc:374] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:45.105255 2017 raft_consensus.cc:729] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65225e65303c4788b4c52ebb6c1b4db4, State: Initialized, Role: FOLLOWER
I20260924 04:46:45.105676 2018 consensus_queue.cc:260] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } }
I20260924 04:46:45.105715 2017 consensus_queue.cc:260] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } }
I20260924 04:46:45.106530 1993 heartbeater.cc:503] Master 127.0.98.62:41415 was elected leader, sending a full tablet report...
I20260924 04:46:45.106570 1853 heartbeater.cc:503] Master 127.0.98.62:41415 was elected leader, sending a full tablet report...
I20260924 04:46:45.106678 2017 ts_tablet_manager.cc:1428] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:46:45.107201 2018 ts_tablet_manager.cc:1428] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:46:45.241262 2024 raft_consensus.cc:482] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:46:45.241465 2024 raft_consensus.cc:504] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } }
I20260924 04:46:45.242703 2024 leader_election.cc:288] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 963e47baa4e04d22ad511211727d9977 (127.0.98.1:39869), 65225e65303c4788b4c52ebb6c1b4db4 (127.0.98.3:37711)
I20260924 04:46:45.246145 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.242994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39869 (local address 127.0.98.2:38035)
0924 04:46:45.243111 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:45.243127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:45.243222 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:45.243495 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:45.243498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:45.243515 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:45.243620 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.243625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.244375 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.244378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.245142 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.245150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.245753 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.245756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.245857 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.245874 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:46:45.245934 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:46:45.246008 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 04:46:45.246146 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.243088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38035 (local address 127.0.98.1:39869)
0924 04:46:45.243201 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.243206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.243223 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:46:45.243335 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.243338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.243393 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.243474 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:45.243717 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.244265 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.245257 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.245647 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.245735 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.245799 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.245958 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.246023 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:46:45.246728 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.243152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35245 (local address 127.0.98.3:37711)
0924 04:46:45.243298 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.243303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.243553 (+ 250us) server_negotiation.cc:407] Connection header received
0924 04:46:45.243651 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.243654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.243696 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.243773 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:45.243996 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.244492 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.245803 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.246190 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.246296 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.246352 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.246573 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.246634 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:46:45.246742 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.243009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37711 (local address 127.0.98.2:35245)
0924 04:46:45.243472 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:45.243482 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:45.243561 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:45.243783 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:45.243785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:45.243799 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:45.243907 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.243912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.244597 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.244601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.245691 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.245700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.246383 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.246386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.246465 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.246481 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 04:46:45.246541 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:46:45.246600 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":110,"thread_start_us":125,"threads_started":1}
I20260924 04:46:45.246845 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e443299b09ba4cfc95868cdb5b93b48a" candidate_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "963e47baa4e04d22ad511211727d9977" is_pre_election: true
I20260924 04:46:45.247326 1667 raft_consensus.cc:2454] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bc064ce6f074c3fa2bcc5c92d92a938 in term 0.
I20260924 04:46:45.247442 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e443299b09ba4cfc95868cdb5b93b48a" candidate_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65225e65303c4788b4c52ebb6c1b4db4" is_pre_election: true
I20260924 04:46:45.247730 1947 raft_consensus.cc:2454] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bc064ce6f074c3fa2bcc5c92d92a938 in term 0.
I20260924 04:46:45.247915 1740 leader_election.cc:302] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bc064ce6f074c3fa2bcc5c92d92a938, 963e47baa4e04d22ad511211727d9977; no voters:
I20260924 04:46:45.248199 2024 raft_consensus.cc:2781] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:46:45.248281 2024 raft_consensus.cc:482] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:46:45.248311 2024 raft_consensus.cc:3037] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:45.249389 2024 raft_consensus.cc:504] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } }
I20260924 04:46:45.249768 2024 leader_election.cc:288] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [CANDIDATE]: Term 1 election: Requested vote from peers 963e47baa4e04d22ad511211727d9977 (127.0.98.1:39869), 65225e65303c4788b4c52ebb6c1b4db4 (127.0.98.3:37711)
I20260924 04:46:45.250164 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e443299b09ba4cfc95868cdb5b93b48a" candidate_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65225e65303c4788b4c52ebb6c1b4db4"
I20260924 04:46:45.250294 1947 raft_consensus.cc:3037] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:45.250300 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e443299b09ba4cfc95868cdb5b93b48a" candidate_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "963e47baa4e04d22ad511211727d9977"
I20260924 04:46:45.250442 1667 raft_consensus.cc:3037] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:45.251946 1947 raft_consensus.cc:2454] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bc064ce6f074c3fa2bcc5c92d92a938 in term 1.
I20260924 04:46:45.252281 1741 leader_election.cc:302] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bc064ce6f074c3fa2bcc5c92d92a938, 65225e65303c4788b4c52ebb6c1b4db4; no voters:
I20260924 04:46:45.252388 1667 raft_consensus.cc:2454] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bc064ce6f074c3fa2bcc5c92d92a938 in term 1.
I20260924 04:46:45.252524 2024 raft_consensus.cc:2781] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:45.252779 2024 raft_consensus.cc:686] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 1 LEADER]: Becoming Leader. State: Replica: 3bc064ce6f074c3fa2bcc5c92d92a938, State: Running, Role: LEADER
I20260924 04:46:45.253141 2024 consensus_queue.cc:237] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } }
I20260924 04:46:45.256202 1510 catalog_manager.cc:5495] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bc064ce6f074c3fa2bcc5c92d92a938 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } health_report { overall_health: HEALTHY } } }
I20260924 04:46:45.275563 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.272066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43812 (local address 127.0.98.2:33481)
0924 04:46:45.272193 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.272197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.272216 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:46:45.272415 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.272417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.272473 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.272550 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:46:45.272749 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.273289 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.274095 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.274316 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.274394 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.274694 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:46:45.275193 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:46:45.275289 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.275373 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.275423 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 04:46:45.277679 1994 tablet.cc:2369] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:46:45.281332 1947 raft_consensus.cc:1261] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4 [term 1 FOLLOWER]: Refusing update from remote peer 3bc064ce6f074c3fa2bcc5c92d92a938: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:45.281343 1667 raft_consensus.cc:1261] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977 [term 1 FOLLOWER]: Refusing update from remote peer 3bc064ce6f074c3fa2bcc5c92d92a938: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:45.281975 2024 consensus_queue.cc:1035] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:45.282158 2026 consensus_queue.cc:1035] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:45.303061 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:46:45.303256 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:46:45.305806 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d: Sending DeleteTablet for 3 replicas of tablet e443299b09ba4cfc95868cdb5b93b48a
I20260924 04:46:45.306602 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet e443299b09ba4cfc95868cdb5b93b48a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34940
I20260924 04:46:45.306600 1640 tablet_service.cc:1514] Processing DeleteTablet for tablet e443299b09ba4cfc95868cdb5b93b48a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59304
I20260924 04:46:45.306715 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet e443299b09ba4cfc95868cdb5b93b48a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43798
I20260924 04:46:45.307137 2039 tablet_replica.cc:330] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4: stopping tablet replica
I20260924 04:46:45.307430 2039 raft_consensus.cc:2229] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:46:45.307466 2041 tablet_replica.cc:330] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938: stopping tablet replica
I20260924 04:46:45.307749 2039 raft_consensus.cc:2258] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:45.307894 2041 raft_consensus.cc:2229] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:46:45.308120 2040 tablet_replica.cc:330] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977: stopping tablet replica
I20260924 04:46:45.308437 2041 raft_consensus.cc:2258] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:45.308485 2040 raft_consensus.cc:2229] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:46:45.308796 2040 raft_consensus.cc:2258] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:45.313158 2041 ts_tablet_manager.cc:1905] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:45.313210 2039 ts_tablet_manager.cc:1905] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:45.315939 2041 ts_tablet_manager.cc:1918] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:46:45.315963 2040 ts_tablet_manager.cc:1905] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:45.316138 2041 log.cc:1196] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/e443299b09ba4cfc95868cdb5b93b48a
I20260924 04:46:45.316041 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:46:45.316500 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:46:45.316615 2041 ts_tablet_manager.cc:1939] T e443299b09ba4cfc95868cdb5b93b48a P 3bc064ce6f074c3fa2bcc5c92d92a938: Deleting consensus metadata
I20260924 04:46:45.317793 1497 catalog_manager.cc:4841] TS 3bc064ce6f074c3fa2bcc5c92d92a938 (127.0.98.2:33481): tablet e443299b09ba4cfc95868cdb5b93b48a (table test-table [id=43bbd9ac68cd4917933e431620cef5f0]) successfully deleted
I20260924 04:46:45.319875 2039 ts_tablet_manager.cc:1918] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:46:45.319971 2039 log.cc:1196] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/e443299b09ba4cfc95868cdb5b93b48a
I20260924 04:46:45.320264 2039 ts_tablet_manager.cc:1939] T e443299b09ba4cfc95868cdb5b93b48a P 65225e65303c4788b4c52ebb6c1b4db4: Deleting consensus metadata
I20260924 04:46:45.321544 2040 ts_tablet_manager.cc:1918] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:46:45.321575 1640 tablet_service.cc:1467] Processing CreateTablet for tablet 35fbb52885db4e66833d754db99cad49 (DEFAULT_TABLE table=test-table [id=b41c44f34b624fd18e57bff1d332fff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:45.321695 2040 log.cc:1196] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/wal/wals/e443299b09ba4cfc95868cdb5b93b48a
I20260924 04:46:45.321823 1640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fbb52885db4e66833d754db99cad49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:45.321974 2040 ts_tablet_manager.cc:1939] T e443299b09ba4cfc95868cdb5b93b48a P 963e47baa4e04d22ad511211727d9977: Deleting consensus metadata
I20260924 04:46:45.321975 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 35fbb52885db4e66833d754db99cad49 (DEFAULT_TABLE table=test-table [id=b41c44f34b624fd18e57bff1d332fff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:45.322199 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fbb52885db4e66833d754db99cad49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:45.323052 1496 catalog_manager.cc:4841] TS 963e47baa4e04d22ad511211727d9977 (127.0.98.1:39869): tablet e443299b09ba4cfc95868cdb5b93b48a (table test-table [id=43bbd9ac68cd4917933e431620cef5f0]) successfully deleted
I20260924 04:46:45.323215 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 35fbb52885db4e66833d754db99cad49 (DEFAULT_TABLE table=test-table [id=b41c44f34b624fd18e57bff1d332fff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:45.323388 1497 catalog_manager.cc:4841] TS 65225e65303c4788b4c52ebb6c1b4db4 (127.0.98.3:37711): tablet e443299b09ba4cfc95868cdb5b93b48a (table test-table [id=43bbd9ac68cd4917933e431620cef5f0]) successfully deleted
I20260924 04:46:45.323458 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fbb52885db4e66833d754db99cad49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:45.325498 2017 tablet_bootstrap.cc:492] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4: Bootstrap starting.
I20260924 04:46:45.325536 2016 tablet_bootstrap.cc:492] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977: Bootstrap starting.
I20260924 04:46:45.326411 2016 tablet_bootstrap.cc:654] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:45.326418 2017 tablet_bootstrap.cc:654] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:45.327816 2016 tablet_bootstrap.cc:492] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977: No bootstrap required, opened a new log
I20260924 04:46:45.327898 2016 ts_tablet_manager.cc:1397] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:46:45.328305 2016 raft_consensus.cc:348] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.328419 2016 raft_consensus.cc:374] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:45.328449 2016 raft_consensus.cc:729] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 963e47baa4e04d22ad511211727d9977, State: Initialized, Role: FOLLOWER
I20260924 04:46:45.328560 2016 consensus_queue.cc:260] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.328692 2018 tablet_bootstrap.cc:492] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938: Bootstrap starting.
I20260924 04:46:45.328805 2016 ts_tablet_manager.cc:1428] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:46:45.329586 2018 tablet_bootstrap.cc:654] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:45.329984 2017 tablet_bootstrap.cc:492] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4: No bootstrap required, opened a new log
I20260924 04:46:45.330061 2017 ts_tablet_manager.cc:1397] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 04:46:45.330490 2017 raft_consensus.cc:348] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.330592 2017 raft_consensus.cc:374] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:45.330621 2017 raft_consensus.cc:729] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65225e65303c4788b4c52ebb6c1b4db4, State: Initialized, Role: FOLLOWER
I20260924 04:46:45.330730 2018 tablet_bootstrap.cc:492] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938: No bootstrap required, opened a new log
I20260924 04:46:45.330734 2017 consensus_queue.cc:260] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.330808 2018 ts_tablet_manager.cc:1397] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:46:45.330998 2017 ts_tablet_manager.cc:1428] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:46:45.331259 2018 raft_consensus.cc:348] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.331363 2018 raft_consensus.cc:374] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:45.331400 2018 raft_consensus.cc:729] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc064ce6f074c3fa2bcc5c92d92a938, State: Initialized, Role: FOLLOWER
I20260924 04:46:45.331506 2018 consensus_queue.cc:260] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.331725 2018 ts_tablet_manager.cc:1428] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260924 04:46:45.337742 1714 tablet.cc:2369] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:46:45.516562 2034 raft_consensus.cc:482] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:46:45.516728 2034 raft_consensus.cc:504] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.517194 2034 leader_election.cc:288] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 963e47baa4e04d22ad511211727d9977 (127.0.98.1:39869), 65225e65303c4788b4c52ebb6c1b4db4 (127.0.98.3:37711)
I20260924 04:46:45.517668 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fbb52885db4e66833d754db99cad49" candidate_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "963e47baa4e04d22ad511211727d9977" is_pre_election: true
I20260924 04:46:45.517673 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fbb52885db4e66833d754db99cad49" candidate_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65225e65303c4788b4c52ebb6c1b4db4" is_pre_election: true
I20260924 04:46:45.517805 1667 raft_consensus.cc:2454] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bc064ce6f074c3fa2bcc5c92d92a938 in term 0.
I20260924 04:46:45.517814 1947 raft_consensus.cc:2454] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bc064ce6f074c3fa2bcc5c92d92a938 in term 0.
I20260924 04:46:45.518105 1740 leader_election.cc:302] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bc064ce6f074c3fa2bcc5c92d92a938, 963e47baa4e04d22ad511211727d9977; no voters:
I20260924 04:46:45.518296 2034 raft_consensus.cc:2781] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:46:45.518362 2034 raft_consensus.cc:482] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:46:45.518393 2034 raft_consensus.cc:3037] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:45.519462 2034 raft_consensus.cc:504] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.519778 2034 leader_election.cc:288] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [CANDIDATE]: Term 1 election: Requested vote from peers 963e47baa4e04d22ad511211727d9977 (127.0.98.1:39869), 65225e65303c4788b4c52ebb6c1b4db4 (127.0.98.3:37711)
I20260924 04:46:45.520121 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fbb52885db4e66833d754db99cad49" candidate_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "963e47baa4e04d22ad511211727d9977"
I20260924 04:46:45.520149 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fbb52885db4e66833d754db99cad49" candidate_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65225e65303c4788b4c52ebb6c1b4db4"
I20260924 04:46:45.520222 1667 raft_consensus.cc:3037] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:45.520264 1947 raft_consensus.cc:3037] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:45.521101 1667 raft_consensus.cc:2454] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bc064ce6f074c3fa2bcc5c92d92a938 in term 1.
I20260924 04:46:45.521162 1947 raft_consensus.cc:2454] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bc064ce6f074c3fa2bcc5c92d92a938 in term 1.
I20260924 04:46:45.521368 1740 leader_election.cc:302] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bc064ce6f074c3fa2bcc5c92d92a938, 963e47baa4e04d22ad511211727d9977; no voters:
I20260924 04:46:45.521500 2034 raft_consensus.cc:2781] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:45.521584 2034 raft_consensus.cc:686] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 1 LEADER]: Becoming Leader. State: Replica: 3bc064ce6f074c3fa2bcc5c92d92a938, State: Running, Role: LEADER
I20260924 04:46:45.521713 2034 consensus_queue.cc:237] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.523331 1510 catalog_manager.cc:5495] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bc064ce6f074c3fa2bcc5c92d92a938 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } health_report { overall_health: UNKNOWN } } }
W20260924 04:46:45.553527 1854 tablet.cc:2369] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:46:45 dist-test-slave-hvcl krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:45.601711 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.592586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41415 (local address 127.0.0.1:39332)
0924 04:46:45.592833 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:45.592849 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:45.592949 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:45.593197 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:45.593199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:45.593588 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:45.593680 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.593686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.594769 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.594771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.595396 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.595405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.595947 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.595949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.596010 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.596573 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:45.596585 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:45.598248 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:45.600104 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:45.600112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:45.600117 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:45.600362 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:45.600638 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:45.600641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:45.600643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:45.600723 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:45.601187 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:45.601193 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:45.601299 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 04:46:45.601390 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 04:46:45.601538 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260924 04:46:45.601742 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.592707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39332 (local address 127.0.98.62:41415)
0924 04:46:45.592918 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.592922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.592938 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:46:45.593036 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.593039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.593083 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.593166 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:45.593778 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.594661 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.595490 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.595841 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.595944 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.598420 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:45.598424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:45.598426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:45.598450 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:45.600000 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:45.600446 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:45.600449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:45.600450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:45.600532 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:45.600828 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:45.600831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:45.600833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:45.601046 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:45.601134 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.601461 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.601601 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":140,"thread_start_us":59,"threads_started":1}
I20260924 04:46:45.605641 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39332:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:46:45.606343 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:46:45.611342 1787 tablet_service.cc:1467] Processing CreateTablet for tablet be58f93e5c384987a0765096d456120c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8cc506cc54a84777a3768860c0e54bfd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:46:45.611334 1640 tablet_service.cc:1467] Processing CreateTablet for tablet be58f93e5c384987a0765096d456120c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8cc506cc54a84777a3768860c0e54bfd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:46:45.611416 1927 tablet_service.cc:1467] Processing CreateTablet for tablet be58f93e5c384987a0765096d456120c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8cc506cc54a84777a3768860c0e54bfd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:46:45.611732 1640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be58f93e5c384987a0765096d456120c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:45.611732 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be58f93e5c384987a0765096d456120c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:45.611747 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be58f93e5c384987a0765096d456120c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:45.614102 2018 tablet_bootstrap.cc:492] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938: Bootstrap starting.
I20260924 04:46:45.614671 2016 tablet_bootstrap.cc:492] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977: Bootstrap starting.
I20260924 04:46:45.614671 2017 tablet_bootstrap.cc:492] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4: Bootstrap starting.
I20260924 04:46:45.615010 2018 tablet_bootstrap.cc:654] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:45.615602 2016 tablet_bootstrap.cc:654] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:45.615623 2017 tablet_bootstrap.cc:654] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:45.616000 2018 tablet_bootstrap.cc:492] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938: No bootstrap required, opened a new log
I20260924 04:46:45.616078 2018 ts_tablet_manager.cc:1397] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:46:45.616470 2018 raft_consensus.cc:348] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.616590 2018 raft_consensus.cc:374] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:45.616621 2018 raft_consensus.cc:729] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc064ce6f074c3fa2bcc5c92d92a938, State: Initialized, Role: FOLLOWER
I20260924 04:46:45.616780 2017 tablet_bootstrap.cc:492] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4: No bootstrap required, opened a new log
I20260924 04:46:45.616731 2018 consensus_queue.cc:260] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.616846 2017 ts_tablet_manager.cc:1397] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:46:45.616927 2016 tablet_bootstrap.cc:492] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977: No bootstrap required, opened a new log
I20260924 04:46:45.616998 2018 ts_tablet_manager.cc:1428] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:46:45.617004 2016 ts_tablet_manager.cc:1397] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:46:45.617194 2034 tablet_replica.cc:439] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } }
I20260924 04:46:45.617247 2017 raft_consensus.cc:348] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.617313 2034 tablet_replica.cc:442] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:46:45.617338 2017 raft_consensus.cc:374] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:45.617367 2017 raft_consensus.cc:729] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65225e65303c4788b4c52ebb6c1b4db4, State: Initialized, Role: FOLLOWER
I20260924 04:46:45.617367 2016 raft_consensus.cc:348] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.617450 2016 raft_consensus.cc:374] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:45.617476 2016 raft_consensus.cc:729] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 963e47baa4e04d22ad511211727d9977, State: Initialized, Role: FOLLOWER
I20260924 04:46:45.617488 2017 consensus_queue.cc:260] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.617586 2016 consensus_queue.cc:260] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.617738 2017 ts_tablet_manager.cc:1428] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:46:45.617834 2016 ts_tablet_manager.cc:1428] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:46:45.617866 2022 tablet_replica.cc:439] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } }
I20260924 04:46:45.617996 2022 tablet_replica.cc:442] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:46:45.617698 2023 tablet_replica.cc:439] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } }
I20260924 04:46:45.618333 2023 tablet_replica.cc:442] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:46:45.674015 2023 raft_consensus.cc:482] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:46:45.674160 2023 raft_consensus.cc:504] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.675454 2023 leader_election.cc:288] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 963e47baa4e04d22ad511211727d9977 (127.0.98.1:39869), 3bc064ce6f074c3fa2bcc5c92d92a938 (127.0.98.2:33481)
I20260924 04:46:45.680298 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.675759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54109 (local address 127.0.98.1:39869)
0924 04:46:45.675962 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.675966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.676147 (+ 181us) server_negotiation.cc:407] Connection header received
0924 04:46:45.676217 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.676220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.676271 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.676374 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:45.676668 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.677267 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.678901 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.679561 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.679686 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.679748 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.680086 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.680164 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260924 04:46:45.680298 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.675797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48285 (local address 127.0.98.2:33481)
0924 04:46:45.675922 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.675926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.676355 (+ 429us) server_negotiation.cc:407] Connection header received
0924 04:46:45.676473 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.676476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.676520 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.676614 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:45.676855 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.677373 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.678901 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.679561 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.679684 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.679751 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.680086 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.680164 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:46:45.680291 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.675683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33481 (local address 127.0.98.3:48285)
0924 04:46:45.676266 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:45.676280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:45.676376 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:45.676630 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:45.676632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:45.676649 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:45.676754 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.676760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.677505 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.677508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.678773 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.678784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.679711 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.679716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.679924 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.679943 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:46:45.680010 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:46:45.680107 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":73,"thread_start_us":65,"threads_started":1}
I20260924 04:46:45.680298 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.675640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39869 (local address 127.0.98.3:54109)
0924 04:46:45.676040 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:45.676051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:45.676141 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:45.676419 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:45.676421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:45.676438 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:45.676579 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.676586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.677390 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.677394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.678773 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.678784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.679711 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.679716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.679872 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.679901 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 04:46:45.680006 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 04:46:45.680107 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":103,"thread_start_us":93,"threads_started":1}
I20260924 04:46:45.680922 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be58f93e5c384987a0765096d456120c" candidate_uuid: "65225e65303c4788b4c52ebb6c1b4db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "963e47baa4e04d22ad511211727d9977" is_pre_election: true
I20260924 04:46:45.681088 1667 raft_consensus.cc:2454] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65225e65303c4788b4c52ebb6c1b4db4 in term 0.
I20260924 04:46:45.681284 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be58f93e5c384987a0765096d456120c" candidate_uuid: "65225e65303c4788b4c52ebb6c1b4db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" is_pre_election: true
I20260924 04:46:45.681429 1880 leader_election.cc:302] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65225e65303c4788b4c52ebb6c1b4db4, 963e47baa4e04d22ad511211727d9977; no voters:
I20260924 04:46:45.681520 1807 raft_consensus.cc:2454] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65225e65303c4788b4c52ebb6c1b4db4 in term 0.
I20260924 04:46:45.681704 2023 raft_consensus.cc:2781] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:46:45.681772 2023 raft_consensus.cc:482] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:46:45.681826 2023 raft_consensus.cc:3037] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:45.682824 2023 raft_consensus.cc:504] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.683188 2023 leader_election.cc:288] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [CANDIDATE]: Term 1 election: Requested vote from peers 963e47baa4e04d22ad511211727d9977 (127.0.98.1:39869), 3bc064ce6f074c3fa2bcc5c92d92a938 (127.0.98.2:33481)
I20260924 04:46:45.683554 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be58f93e5c384987a0765096d456120c" candidate_uuid: "65225e65303c4788b4c52ebb6c1b4db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "963e47baa4e04d22ad511211727d9977"
I20260924 04:46:45.683687 1667 raft_consensus.cc:3037] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:45.683663 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be58f93e5c384987a0765096d456120c" candidate_uuid: "65225e65303c4788b4c52ebb6c1b4db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938"
I20260924 04:46:45.683774 1807 raft_consensus.cc:3037] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:45.684600 1667 raft_consensus.cc:2454] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65225e65303c4788b4c52ebb6c1b4db4 in term 1.
I20260924 04:46:45.684638 1807 raft_consensus.cc:2454] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65225e65303c4788b4c52ebb6c1b4db4 in term 1.
I20260924 04:46:45.684849 1880 leader_election.cc:302] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65225e65303c4788b4c52ebb6c1b4db4, 963e47baa4e04d22ad511211727d9977; no voters:
I20260924 04:46:45.685026 2023 raft_consensus.cc:2781] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:45.685184 2023 raft_consensus.cc:686] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [term 1 LEADER]: Becoming Leader. State: Replica: 65225e65303c4788b4c52ebb6c1b4db4, State: Running, Role: LEADER
I20260924 04:46:45.685415 2023 consensus_queue.cc:237] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } }
I20260924 04:46:45.686606 2058 tablet_replica.cc:439] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4: TxnStatusTablet state changed. Reason: New leader 65225e65303c4788b4c52ebb6c1b4db4. Latest consensus state: current_term: 1 leader_uuid: "65225e65303c4788b4c52ebb6c1b4db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } }
I20260924 04:46:45.686714 2058 tablet_replica.cc:442] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:46:45.687041 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:46:45.687192 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:46:45.687842 1509 catalog_manager.cc:5495] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65225e65303c4788b4c52ebb6c1b4db4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "65225e65303c4788b4c52ebb6c1b4db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } health_report { overall_health: HEALTHY } } }
I20260924 04:46:45.719175 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.715602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53526 (local address 127.0.98.3:37711)
0924 04:46:45.715716 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.715721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.715735 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:46:45.715910 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.715912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.715968 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.716050 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:46:45.716284 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.716795 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.717658 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.717872 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.717961 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.718244 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:46:45.718674 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:46:45.718864 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.718947 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.719019 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:46:45.719195 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.715476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37711 (local address 127.0.0.1:53526)
0924 04:46:45.715640 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:45.715656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:45.715763 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:45.716076 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:45.716079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:45.716089 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:46:45.716185 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.716190 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.716927 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.716930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.717536 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.717545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.717975 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.717978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.718060 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.718104 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:46:45.718840 (+ 736us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:46:45.718858 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:46:45.718923 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:46:45.719084 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 04:46:45.722097 1667 raft_consensus.cc:1261] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977 [term 1 FOLLOWER]: Refusing update from remote peer 65225e65303c4788b4c52ebb6c1b4db4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:45.722232 1807 raft_consensus.cc:1261] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 1 FOLLOWER]: Refusing update from remote peer 65225e65303c4788b4c52ebb6c1b4db4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:45.722631 2023 consensus_queue.cc:1035] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:45.722844 2058 consensus_queue.cc:1035] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:45.724545 2034 tablet_replica.cc:439] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938: TxnStatusTablet state changed. Reason: New leader 65225e65303c4788b4c52ebb6c1b4db4. Latest consensus state: current_term: 1 leader_uuid: "65225e65303c4788b4c52ebb6c1b4db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } }
I20260924 04:46:45.724548 2022 tablet_replica.cc:439] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977: TxnStatusTablet state changed. Reason: New leader 65225e65303c4788b4c52ebb6c1b4db4. Latest consensus state: current_term: 1 leader_uuid: "65225e65303c4788b4c52ebb6c1b4db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } }
I20260924 04:46:45.724658 2034 tablet_replica.cc:442] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:46:45.724656 2022 tablet_replica.cc:442] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:46:45.726437 2058 tablet_replica.cc:439] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65225e65303c4788b4c52ebb6c1b4db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } }
I20260924 04:46:45.726552 2058 tablet_replica.cc:442] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:46:45.726759 2023 tablet_replica.cc:439] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65225e65303c4788b4c52ebb6c1b4db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } }
I20260924 04:46:45.726845 2023 tablet_replica.cc:442] T be58f93e5c384987a0765096d456120c P 65225e65303c4788b4c52ebb6c1b4db4: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:46:45.727892 2030 mvcc.cc:205] Tried to move back new op lower bound from 7332762442632310784 to 7332762442489704448. Current Snapshot: MvccSnapshot[applied={T|T < 7332762442632310784 or (T in {7332762442632310784})}]
I20260924 04:46:45.728199 2022 tablet_replica.cc:439] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65225e65303c4788b4c52ebb6c1b4db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } }
I20260924 04:46:45.728302 2022 tablet_replica.cc:442] T be58f93e5c384987a0765096d456120c P 963e47baa4e04d22ad511211727d9977: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:46:45.728468 2029 mvcc.cc:205] Tried to move back new op lower bound from 7332762442632310784 to 7332762442489704448. Current Snapshot: MvccSnapshot[applied={T|T < 7332762442632310784}]
I20260924 04:46:45.728772 2034 tablet_replica.cc:439] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65225e65303c4788b4c52ebb6c1b4db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 } } peers { permanent_uuid: "3bc064ce6f074c3fa2bcc5c92d92a938" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 } } }
I20260924 04:46:45.728873 2034 tablet_replica.cc:442] T be58f93e5c384987a0765096d456120c P 3bc064ce6f074c3fa2bcc5c92d92a938: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:46:45.743255 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.739836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56329 (local address 127.0.98.3:37711)
0924 04:46:45.739962 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.739966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.740177 (+ 211us) server_negotiation.cc:407] Connection header received
0924 04:46:45.740272 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.740274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.740322 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.740386 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:46:45.740621 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.741107 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.742292 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.742496 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.742587 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.742850 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:46:45.742917 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:46:45.743019 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.743103 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.743147 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:46:45.743327 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.739720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37711 (local address 127.0.98.2:56329)
0924 04:46:45.740069 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:45.740084 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:45.740190 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:45.740408 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:45.740411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:45.740419 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:46:45.740535 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.740541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.741194 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.741197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.742180 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.742190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.742594 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.742598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.742683 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.742734 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:46:45.743022 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:46:45.743028 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:46:45.743093 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:46:45.743176 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":95,"threads_started":1}
I20260924 04:46:45.749920 1947 raft_consensus.cc:1261] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4 [term 1 FOLLOWER]: Refusing update from remote peer 3bc064ce6f074c3fa2bcc5c92d92a938: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:45.749980 1667 raft_consensus.cc:1261] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977 [term 1 FOLLOWER]: Refusing update from remote peer 3bc064ce6f074c3fa2bcc5c92d92a938: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:45.750329 2034 consensus_queue.cc:1035] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65225e65303c4788b4c52ebb6c1b4db4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:45.750483 2024 consensus_queue.cc:1035] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "963e47baa4e04d22ad511211727d9977" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:45.756220 2032 mvcc.cc:205] Tried to move back new op lower bound from 7332762442747985920 to 7332762441818009600. Current Snapshot: MvccSnapshot[applied={T|T < 7332762442747985920}]
I20260924 04:46:45.773604 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.769648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33481 (local address 127.0.98.3:46045)
0924 04:46:45.770065 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:45.770079 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:45.770213 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:45.770480 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:45.770484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:45.770492 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:46:45.770613 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.770619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.771450 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.771454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.772369 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:45.772380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.772944 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.772947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:45.773019 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.773068 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:46:45.773345 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:46:45.773349 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:46:45.773389 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 04:46:45.773455 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":118,"threads_started":1}
I20260924 04:46:45.773680 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.769763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46045 (local address 127.0.98.2:33481)
0924 04:46:45.769878 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.769883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.770140 (+ 257us) server_negotiation.cc:407] Connection header received
0924 04:46:45.770325 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.770329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.770389 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.770466 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:46:45.770708 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.771355 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.772500 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.772838 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.772932 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.773187 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:46:45.773273 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:46:45.773334 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.773401 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.773528 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:46:45.792433 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:46:45.792579 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:46:45.794617 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4104fd6d5574852b0b91c9bfdc7224d: Sending DeleteTablet for 3 replicas of tablet 35fbb52885db4e66833d754db99cad49
I20260924 04:46:45.795219 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 35fbb52885db4e66833d754db99cad49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43798
I20260924 04:46:45.795222 1640 tablet_service.cc:1514] Processing DeleteTablet for tablet 35fbb52885db4e66833d754db99cad49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59304
I20260924 04:46:45.795440 2041 tablet_replica.cc:330] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938: stopping tablet replica
I20260924 04:46:45.795573 2041 raft_consensus.cc:2229] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:46:45.795670 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 35fbb52885db4e66833d754db99cad49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34940
I20260924 04:46:45.795761 2041 raft_consensus.cc:2258] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:45.795830 2039 tablet_replica.cc:330] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4: stopping tablet replica
I20260924 04:46:45.795928 2039 raft_consensus.cc:2229] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:46:45.795939 2040 tablet_replica.cc:330] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977: stopping tablet replica
I20260924 04:46:45.796031 2039 raft_consensus.cc:2258] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:45.796042 2040 raft_consensus.cc:2229] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:46:45.796145 2040 raft_consensus.cc:2258] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:45.796978 2039 ts_tablet_manager.cc:1905] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:45.797091 2040 ts_tablet_manager.cc:1905] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:45.797253 2041 ts_tablet_manager.cc:1905] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:45.799984 2040 ts_tablet_manager.cc:1918] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:46:45.799999 2041 ts_tablet_manager.cc:1918] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:46:45.800058 2040 log.cc:1196] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-0/wal/wals/35fbb52885db4e66833d754db99cad49
I20260924 04:46:45.800071 2041 log.cc:1196] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/35fbb52885db4e66833d754db99cad49
I20260924 04:46:45.800304 2040 ts_tablet_manager.cc:1939] T 35fbb52885db4e66833d754db99cad49 P 963e47baa4e04d22ad511211727d9977: Deleting consensus metadata
I20260924 04:46:45.800472 2039 ts_tablet_manager.cc:1918] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:46:45.800534 2039 log.cc:1196] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/35fbb52885db4e66833d754db99cad49
I20260924 04:46:45.800741 2039 ts_tablet_manager.cc:1939] T 35fbb52885db4e66833d754db99cad49 P 65225e65303c4788b4c52ebb6c1b4db4: Deleting consensus metadata
I20260924 04:46:45.801716 1497 catalog_manager.cc:4841] TS 65225e65303c4788b4c52ebb6c1b4db4 (127.0.98.3:37711): tablet 35fbb52885db4e66833d754db99cad49 (table test-table [id=b41c44f34b624fd18e57bff1d332fff2]) successfully deleted
I20260924 04:46:45.806348 2041 ts_tablet_manager.cc:1939] T 35fbb52885db4e66833d754db99cad49 P 3bc064ce6f074c3fa2bcc5c92d92a938: Deleting consensus metadata
I20260924 04:46:45.807358 1497 catalog_manager.cc:4841] TS 3bc064ce6f074c3fa2bcc5c92d92a938 (127.0.98.2:33481): tablet 35fbb52885db4e66833d754db99cad49 (table test-table [id=b41c44f34b624fd18e57bff1d332fff2]) successfully deleted
I20260924 04:46:45.808344 1496 catalog_manager.cc:4841] TS 963e47baa4e04d22ad511211727d9977 (127.0.98.1:39869): tablet 35fbb52885db4e66833d754db99cad49 (table test-table [id=b41c44f34b624fd18e57bff1d332fff2]) successfully deleted
Sep 24 04:46:45 dist-test-slave-hvcl krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225205, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:46:45.817880 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.800984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.0.98.1:39869)
0924 04:46:45.801097 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.801102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.806508 (+ 5406us) server_negotiation.cc:407] Connection header received
0924 04:46:45.806765 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.806768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.806820 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.806872 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:45.807957 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.808689 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.809821 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.810158 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.810270 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.813906 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:45.813920 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:45.813935 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:45.813961 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:45.815964 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:45.816478 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:45.816484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:45.816486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:45.816568 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:45.816860 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:45.816863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:45.816865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:45.817161 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:45.817347 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.817580 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.817696 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 04:46:45.818652 1697 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33636
I20260924 04:46:45.831724 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:45.823443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.0.98.62:41415)
0924 04:46:45.823598 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:45.823603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:45.823731 (+ 128us) server_negotiation.cc:407] Connection header received
0924 04:46:45.823837 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:45.823841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:45.823908 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:45.823996 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:45.824632 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.825493 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.826369 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:45.826743 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:45.826854 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:45.828343 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:45.828347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:45.828350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:45.828385 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:45.829909 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:45.830358 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:45.830362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:45.830364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:45.830456 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:45.830701 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:45.830704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:45.830706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:45.830918 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:45.831085 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:45.831421 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 04:46:45.831559 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 04:46:45.832099 1509 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48588
I20260924 04:46:45.833163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 1582
I20260924 04:46:45.840540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 1722
I20260924 04:46:45.846514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 1862
I20260924 04:46:45.852010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 1478
2026-09-24T04:46:45Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5775 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:46:45 dist-test-slave-hvcl krb5kdc[2080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:46:45 dist-test-slave-hvcl krb5kdc[2080](info): set up 2 sockets
Sep 24 04:46:45 dist-test-slave-hvcl krb5kdc[2080](info): commencing operation
krb5kdc: starting...
W20260924 04:46:46.810509 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:46:46 dist-test-slave-hvcl krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:46:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:46:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:46:46.929672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40401
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:47.036348 2093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:47.036638 2093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:47.036693 2093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:47.040721 2093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:46:47.040789 2093 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:46:47.040812 2093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:47.040834 2093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:47.040856 2093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:46:47.040884 2093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:46:47.046211 2093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40401
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:47.047312 2093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:47.048336 2093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:47.056656 2099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:47.056737 2093 server_base.cc:1034] running on GCE node
W20260924 04:46:47.056670 2098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:47.056670 2101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:47.057576 2093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:47.058694 2093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:47.059856 2093 hybrid_clock.cc:648] HybridClock initialized: now 1790225207059827 us; error 40 us; skew 500 ppm
Sep 24 04:46:47 dist-test-slave-hvcl krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:47.062608 2093 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:46:47.063494 2093 webserver.cc:460] Webserver started at http://127.0.98.62:37913/ using document root <none> and password file <none>
I20260924 04:46:47.064020 2093 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:47.064083 2093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:47.064276 2093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:47.066257 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "5c128ab0f054431d9b151ce18bc4352b"
format_stamp: "Formatted at 2026-09-24 04:46:47 on dist-test-slave-hvcl"
server_key: "0018599b90ffa17d643219f1dfe99fd8"
server_key_iv: "30a2a9aab5a535670d1f0e5010752b8f"
server_key_version: "encryptionkey@0"
I20260924 04:46:47.066785 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c128ab0f054431d9b151ce18bc4352b"
format_stamp: "Formatted at 2026-09-24 04:46:47 on dist-test-slave-hvcl"
server_key: "0018599b90ffa17d643219f1dfe99fd8"
server_key_iv: "30a2a9aab5a535670d1f0e5010752b8f"
server_key_version: "encryptionkey@0"
I20260924 04:46:47.070441 2093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:46:47.072970 2108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:47.074234 2093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:46:47.074385 2093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "5c128ab0f054431d9b151ce18bc4352b"
format_stamp: "Formatted at 2026-09-24 04:46:47 on dist-test-slave-hvcl"
server_key: "0018599b90ffa17d643219f1dfe99fd8"
server_key_iv: "30a2a9aab5a535670d1f0e5010752b8f"
server_key_version: "encryptionkey@0"
I20260924 04:46:47.074484 2093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:47.092428 2093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:47.095606 2093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:47.095782 2093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:47.103103 2093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40401
I20260924 04:46:47.103170 2170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40401 every 8 connection(s)
I20260924 04:46:47.104053 2093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:46:47.105700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 2093
I20260924 04:46:47.105836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:46:47.106102 392 external_mini_cluster.cc:1428] Setting key 2a3273b1bad58b574e1833dbf5c3b5f2
I20260924 04:46:47.107118 2171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:47.112640 2171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b: Bootstrap starting.
Sep 24 04:46:47 dist-test-slave-hvcl krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:47.115006 2171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:47.115701 2171 log.cc:824] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:47.117591 2171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b: No bootstrap required, opened a new log
I20260924 04:46:47.119881 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.107547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.0.98.62:40401)
0924 04:46:47.107994 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:47.108009 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:47.108045 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:46:47.108669 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:47.108684 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:47.109026 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:47.109330 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:47.110088 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.111105 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.111925 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.112387 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.112526 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.114993 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:47.115000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:47.115018 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:47.115048 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:47.117273 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.117698 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.117703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.117708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.117811 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.118061 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.118064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.118066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.118440 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:47.118615 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:47.118888 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 04:46:47.119185 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":130,"threads_started":1}
I20260924 04:46:47.120713 2171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c128ab0f054431d9b151ce18bc4352b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40401 } }
I20260924 04:46:47.121011 2171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:47.121071 2171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c128ab0f054431d9b151ce18bc4352b, State: Initialized, Role: FOLLOWER
I20260924 04:46:47.121604 2171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c128ab0f054431d9b151ce18bc4352b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40401 } }
I20260924 04:46:47.121762 2171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:46:47.121824 2171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:46:47.121918 2171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:47.122828 2171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c128ab0f054431d9b151ce18bc4352b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40401 } }
I20260924 04:46:47.123193 2171 leader_election.cc:302] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c128ab0f054431d9b151ce18bc4352b; no voters:
I20260924 04:46:47.123508 2171 leader_election.cc:288] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:46:47.124826 2171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:46:47.130414 2188 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:46:47.130861 2188 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:46:47.130571 2176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:47.131234 2176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [term 1 LEADER]: Becoming Leader. State: Replica: 5c128ab0f054431d9b151ce18bc4352b, State: Running, Role: LEADER
I20260924 04:46:47.131605 2176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c128ab0f054431d9b151ce18bc4352b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40401 } }
I20260924 04:46:47.133702 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c128ab0f054431d9b151ce18bc4352b. Latest consensus state: current_term: 1 leader_uuid: "5c128ab0f054431d9b151ce18bc4352b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c128ab0f054431d9b151ce18bc4352b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40401 } } }
I20260924 04:46:47.133733 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c128ab0f054431d9b151ce18bc4352b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c128ab0f054431d9b151ce18bc4352b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40401 } } }
I20260924 04:46:47.133821 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:47.133841 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:47.134176 2192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:46:47.135040 2192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:46:47.140480 2192 catalog_manager.cc:1347] Generated new cluster ID: a9c1f3870f1d4027b092af287e625c7a
I20260924 04:46:47.140547 2192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:46:47.150681 2192 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:46:47.151973 2192 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:46:47.163012 2192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b: Generated new TSK 0
I20260924 04:46:47.163723 2192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:46:47.202227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40401
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:47.312347 2197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:47.312623 2197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:47.312690 2197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:47.316756 2197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:47.316826 2197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:47.316882 2197 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:46:47.316906 2197 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:46:47.316962 2197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:46:47.322311 2197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:47.323449 2197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:47.324512 2197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:47.334869 2202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:47.334918 2203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:47.334869 2205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:47.335471 2197 server_base.cc:1034] running on GCE node
I20260924 04:46:47.335880 2197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:47.336570 2197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:47.337738 2197 hybrid_clock.cc:648] HybridClock initialized: now 1790225207337711 us; error 39 us; skew 500 ppm
Sep 24 04:46:47 dist-test-slave-hvcl krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:47.340523 2197 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:46:47.341421 2197 webserver.cc:460] Webserver started at http://127.0.98.1:39943/ using document root <none> and password file <none>
I20260924 04:46:47.341952 2197 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:47.342013 2197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:47.342195 2197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:47.344249 2197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "082d2b30eb4e424893f0e25b046c73f3"
format_stamp: "Formatted at 2026-09-24 04:46:47 on dist-test-slave-hvcl"
server_key: "7a1c46c1aae0aded99a34507bc790775"
server_key_iv: "ee5b0a76e7ced80595426e5ae7c8d4ee"
server_key_version: "encryptionkey@0"
I20260924 04:46:47.344794 2197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "082d2b30eb4e424893f0e25b046c73f3"
format_stamp: "Formatted at 2026-09-24 04:46:47 on dist-test-slave-hvcl"
server_key: "7a1c46c1aae0aded99a34507bc790775"
server_key_iv: "ee5b0a76e7ced80595426e5ae7c8d4ee"
server_key_version: "encryptionkey@0"
I20260924 04:46:47.348575 2197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:46:47.351099 2212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:47.352439 2197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:46:47.352569 2197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "082d2b30eb4e424893f0e25b046c73f3"
format_stamp: "Formatted at 2026-09-24 04:46:47 on dist-test-slave-hvcl"
server_key: "7a1c46c1aae0aded99a34507bc790775"
server_key_iv: "ee5b0a76e7ced80595426e5ae7c8d4ee"
server_key_version: "encryptionkey@0"
I20260924 04:46:47.352674 2197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:47.390676 2197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:47.393904 2197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:47.394079 2197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:47.395639 2197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:47.395690 2197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:47.395742 2197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:47.395771 2197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:46:47 dist-test-slave-hvcl krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:47.405638 2197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40931
I20260924 04:46:47.405663 2327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40931 every 8 connection(s)
I20260924 04:46:47.406733 2197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:46:47.409118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 2197
I20260924 04:46:47.409232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:46:47.409300 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.396454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34009 (local address 127.0.98.62:40401)
0924 04:46:47.396626 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:47.396631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:47.397459 (+ 828us) server_negotiation.cc:407] Connection header received
0924 04:46:47.398552 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:47.398556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:47.398614 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:47.398695 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:47.400355 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.401137 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.402342 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.402591 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.402687 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.405604 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:47.405608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:47.405611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:47.405637 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:47.407247 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.407836 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.407839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.407841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.407925 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.408254 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.408256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.408257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.408496 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:47.408572 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:47.409037 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 04:46:47.409145 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 04:46:47.409523 392 external_mini_cluster.cc:1428] Setting key 50366ceb80ca87c7b3896f2d96532d5f
I20260924 04:46:47.410295 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.396849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40401 (local address 127.0.98.1:34009)
0924 04:46:47.397297 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:47.397346 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:47.398280 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:47.398919 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:47.398928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:47.399430 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:47.400158 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.400181 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.401274 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.401278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.402225 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.402235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.402735 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.402740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.402954 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.403606 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:47.403629 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:47.405421 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:47.407463 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:47.407468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:47.407483 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:47.407740 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:47.408036 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:47.408038 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:47.408040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:47.408177 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:47.408580 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:47.408585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:47.408814 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 04:46:47.409004 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 04:46:47.409332 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260924 04:46:47.411579 2328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40401
I20260924 04:46:47.411729 2328 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:47.412252 2328 heartbeater.cc:511] Master 127.0.98.62:40401 requested a full tablet report, sending...
I20260924 04:46:47.413883 2125 ts_manager.cc:194] Registered new tserver with Master: 082d2b30eb4e424893f0e25b046c73f3 (127.0.98.1:40931)
I20260924 04:46:47.415320 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34009
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:46:47.421507 2335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.413690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40401 (local address 127.0.98.1:55763)
0924 04:46:47.413960 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:47.413974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:47.414056 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:47.414277 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:47.414280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:47.414488 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:47.414606 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.414612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.415625 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.415629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.416507 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.416518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.417045 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.417049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.417166 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.417709 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:47.417724 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:47.418131 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:47.419923 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:47.419926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:47.419927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:47.420165 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:47.420451 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:47.420453 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:47.420454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:47.420524 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:47.421005 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:47.421009 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:47.421121 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 04:46:47.421225 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:46:47.421354 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260924 04:46:47.421587 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.413988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55763 (local address 127.0.98.62:40401)
0924 04:46:47.414087 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:47.414091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:47.414103 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:46:47.414137 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:47.414140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:47.414177 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:47.414255 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:47.414724 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.415488 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.416626 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.416929 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.417027 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.418338 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:47.418341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:47.418343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:47.418365 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:47.419779 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.420267 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.420271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.420273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.420370 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.420605 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.420607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.420608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.420770 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:47.420964 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:47.421305 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 04:46:47.421457 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:46:47.438735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40401
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:47.548009 2337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:47.548262 2337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:47.548327 2337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:47.552387 2337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:47.552456 2337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:47.552515 2337 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:46:47.552541 2337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:46:47.552593 2337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:46:47.558027 2337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:47.559126 2337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:47.560133 2337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:47.569545 2343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:47.569569 2342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:47.569674 2345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:47.569773 2337 server_base.cc:1034] running on GCE node
I20260924 04:46:47.570250 2337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:47.570894 2337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:47.572057 2337 hybrid_clock.cc:648] HybridClock initialized: now 1790225207572037 us; error 33 us; skew 500 ppm
Sep 24 04:46:47 dist-test-slave-hvcl krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:47.574581 2337 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:46:47.575410 2337 webserver.cc:460] Webserver started at http://127.0.98.2:34713/ using document root <none> and password file <none>
I20260924 04:46:47.575925 2337 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:47.575990 2337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:47.576169 2337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:47.578195 2337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "e53e00964bd145f08f7b05edab2fc529"
format_stamp: "Formatted at 2026-09-24 04:46:47 on dist-test-slave-hvcl"
server_key: "25e8b809bc7d4f1a34016b97d6c6adf0"
server_key_iv: "652cf40d4912d75052f9b926892775ca"
server_key_version: "encryptionkey@0"
I20260924 04:46:47.578728 2337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e53e00964bd145f08f7b05edab2fc529"
format_stamp: "Formatted at 2026-09-24 04:46:47 on dist-test-slave-hvcl"
server_key: "25e8b809bc7d4f1a34016b97d6c6adf0"
server_key_iv: "652cf40d4912d75052f9b926892775ca"
server_key_version: "encryptionkey@0"
I20260924 04:46:47.582479 2337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:46:47.584892 2352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:47.586072 2337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:46:47.586174 2337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "e53e00964bd145f08f7b05edab2fc529"
format_stamp: "Formatted at 2026-09-24 04:46:47 on dist-test-slave-hvcl"
server_key: "25e8b809bc7d4f1a34016b97d6c6adf0"
server_key_iv: "652cf40d4912d75052f9b926892775ca"
server_key_version: "encryptionkey@0"
I20260924 04:46:47.586264 2337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:47.598681 2337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:47.601714 2337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:47.601897 2337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:47.603389 2337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:47.603447 2337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:47.603502 2337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:47.603526 2337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:46:47 dist-test-slave-hvcl krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:47.613189 2337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34831
I20260924 04:46:47.614050 2337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:46:47.614742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 2337
I20260924 04:46:47.614845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:46:47.615171 392 external_mini_cluster.cc:1428] Setting key 0fc29223965765301e2b41bdfcec87da
I20260924 04:46:47.614748 2467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34831 every 8 connection(s)
I20260924 04:46:47.617964 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.604646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40401 (local address 127.0.98.2:40927)
0924 04:46:47.605080 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:47.605129 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:47.606084 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:47.606777 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:47.606787 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:47.607200 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:47.607778 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.607791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.608640 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.608643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.609621 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.609632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.610118 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.610122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.610288 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.611417 (+ 1129us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:47.611433 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:47.612958 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:47.614774 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:47.614779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:47.614795 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:47.615153 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:47.615471 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:47.615474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:47.615475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:47.615646 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:47.616105 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:47.616110 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:47.616367 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 04:46:47.616577 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 04:46:47.616907 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":110,"threads_started":1}
I20260924 04:46:47.618929 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.604185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40927 (local address 127.0.98.62:40401)
0924 04:46:47.604378 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:47.604383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:47.605252 (+ 869us) server_negotiation.cc:407] Connection header received
0924 04:46:47.606403 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:47.606407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:47.606471 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:47.606569 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:47.607936 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.608522 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.609742 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.609967 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.610056 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.613182 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:47.613186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:47.613188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:47.613213 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:47.614606 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.615262 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.615266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.615268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.615368 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.615760 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.615764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.615765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.615992 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:47.616083 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:47.618650 (+ 2567us) server_negotiation.cc:299] Negotiation successful
0924 04:46:47.618783 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:46:47.619872 2468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40401
I20260924 04:46:47.620023 2468 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:47.620554 2468 heartbeater.cc:511] Master 127.0.98.62:40401 requested a full tablet report, sending...
I20260924 04:46:47.621665 2125 ts_manager.cc:194] Registered new tserver with Master: e53e00964bd145f08f7b05edab2fc529 (127.0.98.2:34831)
I20260924 04:46:47.622421 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40927
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:46:47.632033 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.622127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37493 (local address 127.0.98.62:40401)
0924 04:46:47.622224 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:47.622228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:47.625249 (+ 3021us) server_negotiation.cc:407] Connection header received
0924 04:46:47.625407 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:47.625410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:47.625472 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:47.625537 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:47.625970 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.626546 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.627435 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.627711 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.627804 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.628981 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:47.628985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:47.628988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:47.629015 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:47.630379 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.630839 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.630841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.630843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.630918 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.631199 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.631202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.631204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.631417 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:47.631504 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:47.631749 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 04:46:47.631888 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:46:47.632040 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.621845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40401 (local address 127.0.98.2:37493)
0924 04:46:47.625173 (+ 3328us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:47.625186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:47.625275 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:47.625567 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:47.625569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:47.625724 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:47.625854 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.625860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.626663 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.626667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.627331 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.627340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.627839 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.627843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.627940 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.628468 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:47.628479 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:47.628824 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:47.630492 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:47.630494 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:47.630496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:47.630719 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:47.631030 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:47.631033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:47.631035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:47.631116 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:47.631515 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:47.631518 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:47.631601 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 04:46:47.631685 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:46:47.631829 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3253,"thread_start_us":73,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:46:47.644022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40401
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:47.752038 2477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:47.752300 2477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:47.752347 2477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:47.756353 2477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:47.756418 2477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:47.756458 2477 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:46:47.756480 2477 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:46:47.756533 2477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:46:47.761899 2477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:47.762917 2477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:47.763916 2477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:47.773631 2483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:47.773612 2482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:47.773669 2477 server_base.cc:1034] running on GCE node
W20260924 04:46:47.773597 2485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:47.774286 2477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:47.774909 2477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:47.776058 2477 hybrid_clock.cc:648] HybridClock initialized: now 1790225207776036 us; error 36 us; skew 500 ppm
Sep 24 04:46:47 dist-test-slave-hvcl krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:47.778849 2477 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:46:47.779714 2477 webserver.cc:460] Webserver started at http://127.0.98.3:39029/ using document root <none> and password file <none>
I20260924 04:46:47.780272 2477 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:47.780334 2477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:47.780539 2477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:47.782545 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "b2148181fd1542e7a514940edc036a41"
format_stamp: "Formatted at 2026-09-24 04:46:47 on dist-test-slave-hvcl"
server_key: "80af43930b9cc321afbf145d8092ac16"
server_key_iv: "273b526c05c549cb0864738a367f8fc2"
server_key_version: "encryptionkey@0"
I20260924 04:46:47.783123 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b2148181fd1542e7a514940edc036a41"
format_stamp: "Formatted at 2026-09-24 04:46:47 on dist-test-slave-hvcl"
server_key: "80af43930b9cc321afbf145d8092ac16"
server_key_iv: "273b526c05c549cb0864738a367f8fc2"
server_key_version: "encryptionkey@0"
I20260924 04:46:47.786834 2477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:46:47.789284 2492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:47.790644 2477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:46:47.790745 2477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "b2148181fd1542e7a514940edc036a41"
format_stamp: "Formatted at 2026-09-24 04:46:47 on dist-test-slave-hvcl"
server_key: "80af43930b9cc321afbf145d8092ac16"
server_key_iv: "273b526c05c549cb0864738a367f8fc2"
server_key_version: "encryptionkey@0"
I20260924 04:46:47.790839 2477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:47.836166 2477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:47.839355 2477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:47.839540 2477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:47.840956 2477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:47.841005 2477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:47.841056 2477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:47.841084 2477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:47.850728 2477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32863
I20260924 04:46:47.850788 2607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32863 every 8 connection(s)
Sep 24 04:46:47 dist-test-slave-hvcl krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:47.851801 2477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:46:47.855180 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.841809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54097 (local address 127.0.98.62:40401)
0924 04:46:47.841956 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:47.841960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:47.842814 (+ 854us) server_negotiation.cc:407] Connection header received
0924 04:46:47.843893 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:47.843897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:47.843950 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:47.844033 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:47.845508 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.846315 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.847983 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.848322 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.848441 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.851631 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:47.851635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:47.851638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:47.851667 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:47.853155 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.853686 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.853689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.853695 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.853779 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.854179 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.854181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.854182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.854355 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:47.854426 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:47.854904 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 04:46:47.855047 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:46:47.856045 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.842211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40401 (local address 127.0.98.3:54097)
0924 04:46:47.842660 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:47.842710 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:47.843674 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:47.844226 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:47.844235 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:47.844673 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:47.845342 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.845361 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.846465 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.846469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.847851 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.847861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.848468 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.848472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.848729 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.849472 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:47.849493 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:47.851454 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:47.853261 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:47.853265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:47.853276 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:47.853560 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:47.853901 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:47.853905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:47.853907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:47.854095 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:47.854439 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:47.854445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:47.854675 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 04:46:47.854861 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 04:46:47.855166 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":105,"threads_started":1}
I20260924 04:46:47.857439 2608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40401
I20260924 04:46:47.857590 2608 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:47.858134 2608 heartbeater.cc:511] Master 127.0.98.62:40401 requested a full tablet report, sending...
I20260924 04:46:47.859297 2125 ts_manager.cc:194] Registered new tserver with Master: b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863)
I20260924 04:46:47.859946 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54097
I20260924 04:46:47.860385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 2477
I20260924 04:46:47.860483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:46:47.860720 392 external_mini_cluster.cc:1428] Setting key aa8569b921b6e90b85953e77aab8863c
I20260924 04:46:47.862516 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:46:47.867163 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.859656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40401 (local address 127.0.98.3:36991)
0924 04:46:47.860005 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:47.860019 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:47.860113 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:47.860430 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:47.860433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:47.860639 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:47.860779 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.860786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.861804 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.861807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.862531 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.862541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.863064 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.863068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.863166 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.863634 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:47.863645 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:47.864008 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:47.865610 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:47.865613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:47.865615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:47.865882 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:47.866152 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:47.866155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:47.866158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:47.866249 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:47.866617 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:47.866620 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:47.866720 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 04:46:47.866823 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 04:46:47.866993 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260924 04:46:47.867172 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.859737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36991 (local address 127.0.98.62:40401)
0924 04:46:47.859854 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:47.859859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:47.860075 (+ 216us) server_negotiation.cc:407] Connection header received
0924 04:46:47.860216 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:47.860219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:47.860264 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:47.860307 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:47.860899 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.861569 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.862669 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.862924 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.863059 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.864146 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:47.864149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:47.864151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:47.864173 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:47.865500 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.865975 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.865979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.865981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.866061 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.866325 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.866327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.866329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.866486 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:47.866608 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:47.866887 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 04:46:47.867027 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 04:46:47 dist-test-slave-hvcl krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:46:47 dist-test-slave-hvcl krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:47.887065 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.879312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.0.98.62:40401)
0924 04:46:47.879432 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:47.879435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:47.879446 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:46:47.879535 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:47.879538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:47.879576 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:47.879646 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:47.880348 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.880862 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.881707 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.881916 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.881999 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.884281 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:47.884284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:47.884286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:47.884309 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:47.885607 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.885964 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.885968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.885970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.886038 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:47.886277 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:47.886280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:47.886282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:47.886446 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:47.886562 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:47.886779 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 04:46:47.886885 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:46:47.889791 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:46:47.892793 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:46:47.906199 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.900820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34831 (local address 127.0.0.1:59172)
0924 04:46:47.901231 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:47.901287 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:47.901417 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:47.902192 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:47.902198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:47.902222 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:47.902352 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.902359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.903500 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.903503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.904623 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.904633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.905684 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.905687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.905792 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.905831 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 04:46:47.905935 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:46:47.906046 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
I20260924 04:46:47.906236 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.900957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.0.98.2:34831)
0924 04:46:47.901392 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:47.901406 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:47.901429 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:46:47.901639 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:47.901645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:47.901867 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:47.902019 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:47.902478 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.903414 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.904755 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.905580 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.905683 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.905842 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:47.905986 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 04:46:47.906089 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":131,"threads_started":1}
I20260924 04:46:47.906517 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.900879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40931 (local address 127.0.0.1:33628)
0924 04:46:47.901563 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:47.901576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:47.901642 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:47.902517 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:47.902520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:47.902536 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:47.902625 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.902629 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.904127 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.904131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.905250 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.905260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.906183 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.906186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.906276 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.906298 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:46:47.906348 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 04:46:47.906408 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":73,"threads_started":1}
I20260924 04:46:47.906886 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.901579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32863 (local address 127.0.0.1:51956)
0924 04:46:47.902453 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:47.902465 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:47.902594 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:47.903075 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:47.903078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:47.903096 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:47.903177 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.903182 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.904378 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.904381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.905376 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.905385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.906542 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.906544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.906616 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.906636 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:46:47.906700 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:46:47.906764 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":79,"threads_started":1}
I20260924 04:46:47.907074 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.901705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.0.98.1:40931)
0924 04:46:47.902054 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:47.902068 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:47.902088 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:46:47.902157 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:47.902162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:47.902350 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:47.902764 (+ 414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:47.902883 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.904010 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.905374 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.906065 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.906610 (+ 545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.906770 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:47.906819 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 04:46:47.906917 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":83,"threads_started":1}
I20260924 04:46:47.907282 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.901917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.0.98.3:32863)
0924 04:46:47.902316 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:47.902327 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:47.902532 (+ 205us) server_negotiation.cc:407] Connection header received
0924 04:46:47.902733 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:47.902739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:47.902923 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:47.903086 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:47.903278 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.904275 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.905495 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.906237 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.906335 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.906482 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:47.907052 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 04:46:47.907150 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":116,"threads_started":1}
I20260924 04:46:47.909618 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 60c6c4b6ea3543af9b789c3359520d74 (DEFAULT_TABLE table=test-table [id=48e8c67a95314262a890b81f4ff932a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:47.910723 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c6c4b6ea3543af9b789c3359520d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:47.910398 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 60c6c4b6ea3543af9b789c3359520d74 (DEFAULT_TABLE table=test-table [id=48e8c67a95314262a890b81f4ff932a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:47.911441 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c6c4b6ea3543af9b789c3359520d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:47.911978 2262 tablet_service.cc:1467] Processing CreateTablet for tablet 60c6c4b6ea3543af9b789c3359520d74 (DEFAULT_TABLE table=test-table [id=48e8c67a95314262a890b81f4ff932a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:47.912889 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c6c4b6ea3543af9b789c3359520d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:47.918471 2631 tablet_bootstrap.cc:492] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3: Bootstrap starting.
I20260924 04:46:47.919219 2633 tablet_bootstrap.cc:492] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529: Bootstrap starting.
I20260924 04:46:47.919639 2632 tablet_bootstrap.cc:492] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41: Bootstrap starting.
I20260924 04:46:47.921087 2631 tablet_bootstrap.cc:654] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:47.921433 2632 tablet_bootstrap.cc:654] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:47.921803 2631 log.cc:824] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:47.922116 2632 log.cc:824] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:47.924129 2633 tablet_bootstrap.cc:654] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:47.924365 2632 tablet_bootstrap.cc:492] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41: No bootstrap required, opened a new log
I20260924 04:46:47.924365 2631 tablet_bootstrap.cc:492] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3: No bootstrap required, opened a new log
I20260924 04:46:47.924561 2631 ts_tablet_manager.cc:1397] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:46:47.924561 2632 ts_tablet_manager.cc:1397] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:46:47.924885 2633 log.cc:824] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:47.926715 2633 tablet_bootstrap.cc:492] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529: No bootstrap required, opened a new log
I20260924 04:46:47.926898 2633 ts_tablet_manager.cc:1397] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260924 04:46:47.928499 2631 raft_consensus.cc:348] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:47.928499 2632 raft_consensus.cc:348] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:47.928789 2632 raft_consensus.cc:374] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:47.928789 2631 raft_consensus.cc:374] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:47.928843 2632 raft_consensus.cc:729] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2148181fd1542e7a514940edc036a41, State: Initialized, Role: FOLLOWER
I20260924 04:46:47.928851 2631 raft_consensus.cc:729] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082d2b30eb4e424893f0e25b046c73f3, State: Initialized, Role: FOLLOWER
I20260924 04:46:47.929343 2632 consensus_queue.cc:260] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:47.929343 2631 consensus_queue.cc:260] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:47.929948 2633 raft_consensus.cc:348] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:47.930188 2633 raft_consensus.cc:374] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:47.930241 2633 raft_consensus.cc:729] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53e00964bd145f08f7b05edab2fc529, State: Initialized, Role: FOLLOWER
I20260924 04:46:47.930332 2632 ts_tablet_manager.cc:1428] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:46:47.930332 2631 ts_tablet_manager.cc:1428] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:46:47.930687 2328 heartbeater.cc:503] Master 127.0.98.62:40401 was elected leader, sending a full tablet report...
I20260924 04:46:47.930689 2608 heartbeater.cc:503] Master 127.0.98.62:40401 was elected leader, sending a full tablet report...
I20260924 04:46:47.931603 2633 consensus_queue.cc:260] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:47.932577 2633 ts_tablet_manager.cc:1428] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:46:47.932909 2468 heartbeater.cc:503] Master 127.0.98.62:40401 was elected leader, sending a full tablet report...
I20260924 04:46:47.956802 2637 raft_consensus.cc:482] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:46:47.957008 2637 raft_consensus.cc:504] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:47.958267 2637 leader_election.cc:288] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e53e00964bd145f08f7b05edab2fc529 (127.0.98.2:34831), b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863)
I20260924 04:46:47.962442 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.958704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34715 (local address 127.0.98.2:34831)
0924 04:46:47.958839 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:47.958843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:47.959082 (+ 239us) server_negotiation.cc:407] Connection header received
0924 04:46:47.959347 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:47.959351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:47.959404 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:47.959481 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:47.959764 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.960290 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.961413 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.961927 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.962025 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.962090 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:47.962255 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 04:46:47.962327 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:46:47.962450 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.958617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34831 (local address 127.0.98.1:34715)
0924 04:46:47.958916 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:47.958931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:47.959090 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:47.959513 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:47.959516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:47.959537 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:47.959657 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.959664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.960402 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.960405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.961300 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.961310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.962036 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.962039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.962147 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.962165 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:46:47.962230 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:46:47.962297 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":106,"threads_started":1}
I20260924 04:46:47.962641 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.959213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39101 (local address 127.0.98.3:32863)
0924 04:46:47.959319 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:47.959324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:47.959618 (+ 294us) server_negotiation.cc:407] Connection header received
0924 04:46:47.959722 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:47.959726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:47.959770 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:47.959830 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:47.960073 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.960561 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.961679 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.962130 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.962223 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.962282 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:47.962464 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 04:46:47.962535 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:46:47.962940 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:47.959100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32863 (local address 127.0.98.1:39101)
0924 04:46:47.959530 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:47.959541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:47.959629 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:47.959852 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:47.959855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:47.959870 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:47.959974 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.959980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.960665 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.960668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.961580 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:47.961590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:47.962243 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:47.962247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:47.962361 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:47.962380 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:46:47.962441 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 04:46:47.962504 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":149,"threads_started":1}
I20260924 04:46:47.963212 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c6c4b6ea3543af9b789c3359520d74" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53e00964bd145f08f7b05edab2fc529" is_pre_election: true
I20260924 04:46:47.963506 2422 raft_consensus.cc:2454] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 0.
I20260924 04:46:47.963512 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c6c4b6ea3543af9b789c3359520d74" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2148181fd1542e7a514940edc036a41" is_pre_election: true
I20260924 04:46:47.963762 2562 raft_consensus.cc:2454] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 0.
I20260924 04:46:47.963977 2214 leader_election.cc:302] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082d2b30eb4e424893f0e25b046c73f3, e53e00964bd145f08f7b05edab2fc529; no voters:
I20260924 04:46:47.964258 2637 raft_consensus.cc:2781] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:46:47.964344 2637 raft_consensus.cc:482] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:46:47.964376 2637 raft_consensus.cc:3037] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:47.965389 2637 raft_consensus.cc:504] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:47.965711 2637 leader_election.cc:288] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 election: Requested vote from peers e53e00964bd145f08f7b05edab2fc529 (127.0.98.2:34831), b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863)
I20260924 04:46:47.966078 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c6c4b6ea3543af9b789c3359520d74" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53e00964bd145f08f7b05edab2fc529"
I20260924 04:46:47.966109 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c6c4b6ea3543af9b789c3359520d74" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2148181fd1542e7a514940edc036a41"
I20260924 04:46:47.966192 2422 raft_consensus.cc:3037] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:47.966224 2562 raft_consensus.cc:3037] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:47.967283 2422 raft_consensus.cc:2454] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 1.
I20260924 04:46:47.967285 2562 raft_consensus.cc:2454] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 1.
I20260924 04:46:47.967555 2214 leader_election.cc:302] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082d2b30eb4e424893f0e25b046c73f3, b2148181fd1542e7a514940edc036a41; no voters:
I20260924 04:46:47.967721 2637 raft_consensus.cc:2781] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:47.967949 2637 raft_consensus.cc:686] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [term 1 LEADER]: Becoming Leader. State: Replica: 082d2b30eb4e424893f0e25b046c73f3, State: Running, Role: LEADER
I20260924 04:46:47.968297 2637 consensus_queue.cc:237] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:47.971478 2124 catalog_manager.cc:5495] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 082d2b30eb4e424893f0e25b046c73f3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "082d2b30eb4e424893f0e25b046c73f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } health_report { overall_health: UNKNOWN } } }
Sep 24 04:46:48 dist-test-slave-hvcl krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:48.017181 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:48.008925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.0.98.62:40401)
0924 04:46:48.009035 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:48.009039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:48.009302 (+ 263us) server_negotiation.cc:407] Connection header received
0924 04:46:48.009424 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:48.009427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:48.009463 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:48.009525 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:48.010227 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:48.010742 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:48.011568 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:48.011788 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:48.011871 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:48.014082 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:48.014085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:48.014087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:48.014109 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:48.015435 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:48.015914 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:48.015917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:48.015918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:48.016005 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:48.016276 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:48.016278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:48.016279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:48.016520 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:48.016599 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:48.016910 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 04:46:48.017049 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":37}
I20260924 04:46:48.017182 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:48.008869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40401 (local address 127.0.0.1:37112)
0924 04:46:48.009210 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:48.009222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:48.009338 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:48.009558 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:48.009562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:48.009993 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:48.010105 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:48.010112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:48.010848 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:48.010852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:48.011451 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:48.011459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:48.011888 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:48.011891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:48.011967 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:48.012515 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:48.012527 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:48.013937 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:48.015561 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:48.015570 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:48.015575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:48.015812 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:48.016111 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:48.016113 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:48.016115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:48.016202 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:48.016636 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:48.016643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:48.016754 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 04:46:48.016851 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 04:46:48.016990 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":129,"threads_started":1}
I20260924 04:46:48.021180 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37112:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:46:48.021941 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:46:48.027064 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 64de58a3b65d4915bddfee50f17856eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6755bd2b62594300b5752cffb10eed7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:46:48.027089 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 64de58a3b65d4915bddfee50f17856eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6755bd2b62594300b5752cffb10eed7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:46:48.026932 2262 tablet_service.cc:1467] Processing CreateTablet for tablet 64de58a3b65d4915bddfee50f17856eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6755bd2b62594300b5752cffb10eed7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:46:48.027685 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64de58a3b65d4915bddfee50f17856eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:48.027696 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64de58a3b65d4915bddfee50f17856eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:48.027696 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64de58a3b65d4915bddfee50f17856eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:48.030102 2633 tablet_bootstrap.cc:492] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529: Bootstrap starting.
I20260924 04:46:48.030465 2632 tablet_bootstrap.cc:492] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41: Bootstrap starting.
I20260924 04:46:48.030472 2631 tablet_bootstrap.cc:492] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3: Bootstrap starting.
I20260924 04:46:48.031170 2633 tablet_bootstrap.cc:654] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:48.031441 2631 tablet_bootstrap.cc:654] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:48.031450 2632 tablet_bootstrap.cc:654] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:48.032604 2633 tablet_bootstrap.cc:492] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529: No bootstrap required, opened a new log
I20260924 04:46:48.032692 2633 ts_tablet_manager.cc:1397] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:46:48.032940 2632 tablet_bootstrap.cc:492] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41: No bootstrap required, opened a new log
I20260924 04:46:48.033003 2632 ts_tablet_manager.cc:1397] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:46:48.033066 2633 raft_consensus.cc:348] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.033152 2633 raft_consensus.cc:374] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:48.033170 2633 raft_consensus.cc:729] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53e00964bd145f08f7b05edab2fc529, State: Initialized, Role: FOLLOWER
I20260924 04:46:48.033280 2633 consensus_queue.cc:260] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.033340 2632 raft_consensus.cc:348] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.033504 2632 raft_consensus.cc:374] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:48.033535 2632 raft_consensus.cc:729] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2148181fd1542e7a514940edc036a41, State: Initialized, Role: FOLLOWER
I20260924 04:46:48.033527 2633 ts_tablet_manager.cc:1428] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:46:48.033545 2639 tablet_replica.cc:439] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } }
I20260924 04:46:48.033668 2632 consensus_queue.cc:260] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.033710 2639 tablet_replica.cc:442] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:46:48.033910 2632 ts_tablet_manager.cc:1428] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:46:48.034127 2638 tablet_replica.cc:439] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } }
I20260924 04:46:48.034263 2638 tablet_replica.cc:442] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:46:48.034757 2631 tablet_bootstrap.cc:492] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3: No bootstrap required, opened a new log
I20260924 04:46:48.034853 2631 ts_tablet_manager.cc:1397] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 04:46:48.035363 2631 raft_consensus.cc:348] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.035477 2631 raft_consensus.cc:374] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:48.035506 2631 raft_consensus.cc:729] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082d2b30eb4e424893f0e25b046c73f3, State: Initialized, Role: FOLLOWER
I20260924 04:46:48.035627 2631 consensus_queue.cc:260] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.035874 2637 tablet_replica.cc:439] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } }
I20260924 04:46:48.036027 2637 tablet_replica.cc:442] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:46:48.035874 2631 ts_tablet_manager.cc:1428] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260924 04:46:48.102854 2609 tablet.cc:2369] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:46:48.116209 2469 tablet.cc:2369] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:46:48.122190 2637 raft_consensus.cc:482] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:46:48.122323 2637 raft_consensus.cc:504] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.122726 2637 leader_election.cc:288] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e53e00964bd145f08f7b05edab2fc529 (127.0.98.2:34831), b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863)
I20260924 04:46:48.123227 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64de58a3b65d4915bddfee50f17856eb" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53e00964bd145f08f7b05edab2fc529" is_pre_election: true
I20260924 04:46:48.123370 2422 raft_consensus.cc:2454] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 0.
I20260924 04:46:48.123332 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64de58a3b65d4915bddfee50f17856eb" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2148181fd1542e7a514940edc036a41" is_pre_election: true
I20260924 04:46:48.123557 2562 raft_consensus.cc:2454] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 0.
I20260924 04:46:48.123762 2214 leader_election.cc:302] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082d2b30eb4e424893f0e25b046c73f3, e53e00964bd145f08f7b05edab2fc529; no voters:
I20260924 04:46:48.123979 2637 raft_consensus.cc:2781] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:46:48.124038 2637 raft_consensus.cc:482] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:46:48.124104 2637 raft_consensus.cc:3037] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:48.125185 2637 raft_consensus.cc:504] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.125535 2637 leader_election.cc:288] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 election: Requested vote from peers e53e00964bd145f08f7b05edab2fc529 (127.0.98.2:34831), b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863)
I20260924 04:46:48.125876 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64de58a3b65d4915bddfee50f17856eb" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53e00964bd145f08f7b05edab2fc529"
I20260924 04:46:48.125967 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64de58a3b65d4915bddfee50f17856eb" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2148181fd1542e7a514940edc036a41"
I20260924 04:46:48.126008 2422 raft_consensus.cc:3037] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:48.126083 2562 raft_consensus.cc:3037] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:48.127053 2422 raft_consensus.cc:2454] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 1.
I20260924 04:46:48.127058 2562 raft_consensus.cc:2454] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 1.
I20260924 04:46:48.127302 2214 leader_election.cc:302] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082d2b30eb4e424893f0e25b046c73f3, e53e00964bd145f08f7b05edab2fc529; no voters:
I20260924 04:46:48.127468 2637 raft_consensus.cc:2781] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:48.127549 2637 raft_consensus.cc:686] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [term 1 LEADER]: Becoming Leader. State: Replica: 082d2b30eb4e424893f0e25b046c73f3, State: Running, Role: LEADER
I20260924 04:46:48.127712 2637 consensus_queue.cc:237] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.128314 2642 tablet_replica.cc:439] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3: TxnStatusTablet state changed. Reason: New leader 082d2b30eb4e424893f0e25b046c73f3. Latest consensus state: current_term: 1 leader_uuid: "082d2b30eb4e424893f0e25b046c73f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } }
I20260924 04:46:48.128407 2642 tablet_replica.cc:442] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:46:48.128769 2654 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:46:48.128989 2654 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:46:48.129424 2124 catalog_manager.cc:5495] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 082d2b30eb4e424893f0e25b046c73f3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "082d2b30eb4e424893f0e25b046c73f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } health_report { overall_health: UNKNOWN } } }
W20260924 04:46:48.158059 2329 tablet.cc:2369] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:46:48.164223 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:48.159765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40931 (local address 127.0.0.1:33634)
0924 04:46:48.160008 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:48.160023 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:48.160149 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:48.160489 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:48.160492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:48.160499 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:46:48.160612 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:48.160618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:48.161319 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:48.161322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:48.162369 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:48.162382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:48.163025 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:48.163029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:48.163118 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:48.163169 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:46:48.163958 (+ 789us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:46:48.163969 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 04:46:48.164023 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 04:46:48.164100 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260924 04:46:48.164245 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:48.159891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.0.98.1:40931)
0924 04:46:48.160014 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:48.160017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:48.160093 (+ 76us) server_negotiation.cc:407] Connection header received
0924 04:46:48.160317 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:48.160320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:48.160380 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:48.160466 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:46:48.160720 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:48.161222 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:48.162524 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:48.162880 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:48.163017 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:48.163303 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:46:48.163854 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:46:48.163966 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:48.164039 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 04:46:48.164110 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:46:48.169864 2422 raft_consensus.cc:1261] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Refusing update from remote peer 082d2b30eb4e424893f0e25b046c73f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:48.169872 2562 raft_consensus.cc:1261] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Refusing update from remote peer 082d2b30eb4e424893f0e25b046c73f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:48.170524 2637 consensus_queue.cc:1035] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:48.170712 2642 consensus_queue.cc:1035] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:48.174477 2639 tablet_replica.cc:439] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529: TxnStatusTablet state changed. Reason: New leader 082d2b30eb4e424893f0e25b046c73f3. Latest consensus state: current_term: 1 leader_uuid: "082d2b30eb4e424893f0e25b046c73f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } }
I20260924 04:46:48.174606 2639 tablet_replica.cc:442] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:46:48.174893 2638 tablet_replica.cc:439] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41: TxnStatusTablet state changed. Reason: New leader 082d2b30eb4e424893f0e25b046c73f3. Latest consensus state: current_term: 1 leader_uuid: "082d2b30eb4e424893f0e25b046c73f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } }
I20260924 04:46:48.175009 2638 tablet_replica.cc:442] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:46:48.178220 2637 tablet_replica.cc:439] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "082d2b30eb4e424893f0e25b046c73f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } }
I20260924 04:46:48.178341 2637 tablet_replica.cc:442] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:46:48.180063 2639 tablet_replica.cc:439] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "082d2b30eb4e424893f0e25b046c73f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } }
I20260924 04:46:48.180195 2639 tablet_replica.cc:442] T 64de58a3b65d4915bddfee50f17856eb P e53e00964bd145f08f7b05edab2fc529: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:46:48.180783 2642 tablet_replica.cc:439] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "082d2b30eb4e424893f0e25b046c73f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } }
I20260924 04:46:48.180886 2642 tablet_replica.cc:442] T 64de58a3b65d4915bddfee50f17856eb P 082d2b30eb4e424893f0e25b046c73f3: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:46:48.182426 2659 mvcc.cc:205] Tried to move back new op lower bound from 7332762452656336896 to 7332762452491948032. Current Snapshot: MvccSnapshot[applied={T|T < 7332762452656336896}]
I20260924 04:46:48.184744 2638 tablet_replica.cc:439] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "082d2b30eb4e424893f0e25b046c73f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } } }
I20260924 04:46:48.184867 2638 tablet_replica.cc:442] T 64de58a3b65d4915bddfee50f17856eb P b2148181fd1542e7a514940edc036a41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:46:48.188020 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:48.183996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.0.98.1:40931)
0924 04:46:48.184461 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:48.184465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:48.184476 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:46:48.184515 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:48.184518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:48.184561 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:48.184626 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:46:48.184912 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:48.185693 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:48.186943 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:48.187245 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:48.187328 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:48.187659 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:46:48.187727 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:46:48.187778 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:48.187879 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 04:46:48.187921 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369}
I20260924 04:46:48.198210 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:48.195206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57473 (local address 127.0.98.1:40931)
0924 04:46:48.195428 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:48.195432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:48.195493 (+ 61us) server_negotiation.cc:407] Connection header received
0924 04:46:48.195671 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:48.195673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:48.195709 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:48.195764 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:46:48.195981 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:48.196469 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:48.197256 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:48.197465 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:48.197543 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:48.197802 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:46:48.197872 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:46:48.197927 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:48.198006 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 04:46:48.198055 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":169}
I20260924 04:46:48.198220 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:48.195127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40931 (local address 127.0.98.1:57473)
0924 04:46:48.195407 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:48.195428 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:48.195533 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:48.195779 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:48.195781 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:48.195787 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:46:48.195896 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:48.195901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:48.196557 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:48.196560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:48.197153 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:48.197161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:48.197550 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:48.197553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:48.197633 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:48.197668 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:46:48.197937 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:46:48.197942 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:46:48.197999 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 04:46:48.198085 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260924 04:46:48.204396 2422 raft_consensus.cc:1261] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Refusing update from remote peer 082d2b30eb4e424893f0e25b046c73f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:48.204418 2562 raft_consensus.cc:1261] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Refusing update from remote peer 082d2b30eb4e424893f0e25b046c73f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:48.204797 2664 consensus_queue.cc:1035] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:48.204957 2642 consensus_queue.cc:1035] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:48.212236 2658 mvcc.cc:205] Tried to move back new op lower bound from 7332762452801871872 to 7332762451841044480. Current Snapshot: MvccSnapshot[applied={T|T < 7332762452801871872}]
I20260924 04:46:48.243760 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:46:48.243923 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:46:48.246255 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b: Sending DeleteTablet for 3 replicas of tablet 60c6c4b6ea3543af9b789c3359520d74
I20260924 04:46:48.247053 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c6c4b6ea3543af9b789c3359520d74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59172
I20260924 04:46:48.247033 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c6c4b6ea3543af9b789c3359520d74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33628
I20260924 04:46:48.247123 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c6c4b6ea3543af9b789c3359520d74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51956
I20260924 04:46:48.247665 2675 tablet_replica.cc:330] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3: stopping tablet replica
I20260924 04:46:48.247727 2677 tablet_replica.cc:330] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41: stopping tablet replica
I20260924 04:46:48.247972 2675 raft_consensus.cc:2229] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:46:48.248003 2677 raft_consensus.cc:2229] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:46:48.248245 2677 raft_consensus.cc:2258] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:48.248319 2675 raft_consensus.cc:2258] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:48.248327 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:46:48.248634 2676 tablet_replica.cc:330] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529: stopping tablet replica
W20260924 04:46:48.248725 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:46:48.248909 2676 raft_consensus.cc:2229] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:46:48.249097 2676 raft_consensus.cc:2258] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:48.250535 2675 ts_tablet_manager.cc:1905] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:48.250645 2677 ts_tablet_manager.cc:1905] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:48.253743 2677 ts_tablet_manager.cc:1918] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:46:48.253845 2677 log.cc:1196] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/60c6c4b6ea3543af9b789c3359520d74
I20260924 04:46:48.254158 2677 ts_tablet_manager.cc:1939] T 60c6c4b6ea3543af9b789c3359520d74 P b2148181fd1542e7a514940edc036a41: Deleting consensus metadata
I20260924 04:46:48.254328 2676 ts_tablet_manager.cc:1905] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:48.254675 2402 tablet_service.cc:1467] Processing CreateTablet for tablet bba38efbc1a544d686b61aaaa46ba618 (DEFAULT_TABLE table=test-table [id=9daa79259d3b4736ab591ec1c167b46e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:48.254947 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba38efbc1a544d686b61aaaa46ba618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:48.255686 2542 tablet_service.cc:1467] Processing CreateTablet for tablet bba38efbc1a544d686b61aaaa46ba618 (DEFAULT_TABLE table=test-table [id=9daa79259d3b4736ab591ec1c167b46e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:48.255925 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba38efbc1a544d686b61aaaa46ba618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:48.256732 2675 ts_tablet_manager.cc:1918] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:46:48.256840 2675 log.cc:1196] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/wal/wals/60c6c4b6ea3543af9b789c3359520d74
I20260924 04:46:48.257272 2675 ts_tablet_manager.cc:1939] T 60c6c4b6ea3543af9b789c3359520d74 P 082d2b30eb4e424893f0e25b046c73f3: Deleting consensus metadata
I20260924 04:46:48.257454 2633 tablet_bootstrap.cc:492] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529: Bootstrap starting.
I20260924 04:46:48.258117 2632 tablet_bootstrap.cc:492] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41: Bootstrap starting.
I20260924 04:46:48.258422 2633 tablet_bootstrap.cc:654] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:48.259011 2632 tablet_bootstrap.cc:654] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:48.259371 2112 catalog_manager.cc:4841] TS 082d2b30eb4e424893f0e25b046c73f3 (127.0.98.1:40931): tablet 60c6c4b6ea3543af9b789c3359520d74 (table test-table [id=48e8c67a95314262a890b81f4ff932a7]) successfully deleted
I20260924 04:46:48.259671 2676 ts_tablet_manager.cc:1918] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:46:48.259756 2676 log.cc:1196] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/60c6c4b6ea3543af9b789c3359520d74
I20260924 04:46:48.259735 2110 catalog_manager.cc:4841] TS b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863): tablet 60c6c4b6ea3543af9b789c3359520d74 (table test-table [id=48e8c67a95314262a890b81f4ff932a7]) successfully deleted
I20260924 04:46:48.259997 2676 ts_tablet_manager.cc:1939] T 60c6c4b6ea3543af9b789c3359520d74 P e53e00964bd145f08f7b05edab2fc529: Deleting consensus metadata
I20260924 04:46:48.260566 2262 tablet_service.cc:1467] Processing CreateTablet for tablet bba38efbc1a544d686b61aaaa46ba618 (DEFAULT_TABLE table=test-table [id=9daa79259d3b4736ab591ec1c167b46e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:48.260864 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba38efbc1a544d686b61aaaa46ba618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:48.261312 2110 catalog_manager.cc:4841] TS e53e00964bd145f08f7b05edab2fc529 (127.0.98.2:34831): tablet 60c6c4b6ea3543af9b789c3359520d74 (table test-table [id=48e8c67a95314262a890b81f4ff932a7]) successfully deleted
I20260924 04:46:48.263353 2631 tablet_bootstrap.cc:492] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3: Bootstrap starting.
I20260924 04:46:48.263435 2632 tablet_bootstrap.cc:492] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41: No bootstrap required, opened a new log
I20260924 04:46:48.263525 2632 ts_tablet_manager.cc:1397] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260924 04:46:48.263967 2632 raft_consensus.cc:348] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.264096 2632 raft_consensus.cc:374] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:48.264127 2632 raft_consensus.cc:729] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2148181fd1542e7a514940edc036a41, State: Initialized, Role: FOLLOWER
I20260924 04:46:48.264293 2631 tablet_bootstrap.cc:654] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:48.264289 2633 tablet_bootstrap.cc:492] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529: No bootstrap required, opened a new log
I20260924 04:46:48.264263 2632 consensus_queue.cc:260] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.264375 2633 ts_tablet_manager.cc:1397] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260924 04:46:48.264569 2632 ts_tablet_manager.cc:1428] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:46:48.264833 2633 raft_consensus.cc:348] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.264942 2633 raft_consensus.cc:374] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:48.264973 2633 raft_consensus.cc:729] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53e00964bd145f08f7b05edab2fc529, State: Initialized, Role: FOLLOWER
I20260924 04:46:48.265095 2633 consensus_queue.cc:260] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.265352 2633 ts_tablet_manager.cc:1428] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:46:48.266242 2631 tablet_bootstrap.cc:492] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3: No bootstrap required, opened a new log
I20260924 04:46:48.266338 2631 ts_tablet_manager.cc:1397] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 04:46:48.266813 2631 raft_consensus.cc:348] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.266939 2631 raft_consensus.cc:374] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:48.267016 2631 raft_consensus.cc:729] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082d2b30eb4e424893f0e25b046c73f3, State: Initialized, Role: FOLLOWER
I20260924 04:46:48.267139 2631 consensus_queue.cc:260] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.267409 2631 ts_tablet_manager.cc:1428] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:46:48.397233 2637 raft_consensus.cc:482] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:46:48.397394 2637 raft_consensus.cc:504] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.397856 2637 leader_election.cc:288] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e53e00964bd145f08f7b05edab2fc529 (127.0.98.2:34831), b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863)
I20260924 04:46:48.398335 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba38efbc1a544d686b61aaaa46ba618" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53e00964bd145f08f7b05edab2fc529" is_pre_election: true
I20260924 04:46:48.398337 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba38efbc1a544d686b61aaaa46ba618" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2148181fd1542e7a514940edc036a41" is_pre_election: true
I20260924 04:46:48.398489 2422 raft_consensus.cc:2454] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 0.
I20260924 04:46:48.398489 2561 raft_consensus.cc:2454] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 0.
I20260924 04:46:48.398804 2214 leader_election.cc:302] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082d2b30eb4e424893f0e25b046c73f3, e53e00964bd145f08f7b05edab2fc529; no voters:
I20260924 04:46:48.399011 2637 raft_consensus.cc:2781] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:46:48.399080 2637 raft_consensus.cc:482] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:46:48.399137 2637 raft_consensus.cc:3037] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:48.400473 2637 raft_consensus.cc:504] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.400835 2637 leader_election.cc:288] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 election: Requested vote from peers e53e00964bd145f08f7b05edab2fc529 (127.0.98.2:34831), b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863)
I20260924 04:46:48.401232 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba38efbc1a544d686b61aaaa46ba618" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53e00964bd145f08f7b05edab2fc529"
I20260924 04:46:48.401305 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba38efbc1a544d686b61aaaa46ba618" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2148181fd1542e7a514940edc036a41"
I20260924 04:46:48.401353 2422 raft_consensus.cc:3037] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:48.401439 2561 raft_consensus.cc:3037] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:48.402217 2422 raft_consensus.cc:2454] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 1.
I20260924 04:46:48.402704 2561 raft_consensus.cc:2454] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 1.
I20260924 04:46:48.403019 2214 leader_election.cc:302] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082d2b30eb4e424893f0e25b046c73f3, b2148181fd1542e7a514940edc036a41; no voters:
I20260924 04:46:48.403174 2637 raft_consensus.cc:2781] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:48.403345 2637 raft_consensus.cc:686] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [term 1 LEADER]: Becoming Leader. State: Replica: 082d2b30eb4e424893f0e25b046c73f3, State: Running, Role: LEADER
I20260924 04:46:48.403482 2637 consensus_queue.cc:237] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:48.405242 2124 catalog_manager.cc:5495] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 082d2b30eb4e424893f0e25b046c73f3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "082d2b30eb4e424893f0e25b046c73f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } health_report { overall_health: UNKNOWN } } }
I20260924 04:46:48.432060 2422 raft_consensus.cc:1261] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Refusing update from remote peer 082d2b30eb4e424893f0e25b046c73f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:48.432283 2561 raft_consensus.cc:1261] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Refusing update from remote peer 082d2b30eb4e424893f0e25b046c73f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:48.432451 2673 consensus_queue.cc:1035] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:48.432945 2673 consensus_queue.cc:1035] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:48.436872 2657 mvcc.cc:205] Tried to move back new op lower bound from 7332762453734428672 to 7332762453622206464. Current Snapshot: MvccSnapshot[applied={T|T < 7332762453734428672}]
W20260924 04:46:48.446391 2657 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 04:46:49.996623 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:46:49.996790 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:46:49.999475 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b: Sending DeleteTablet for 3 replicas of tablet bba38efbc1a544d686b61aaaa46ba618
I20260924 04:46:50.000115 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet bba38efbc1a544d686b61aaaa46ba618 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33628
I20260924 04:46:50.000149 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet bba38efbc1a544d686b61aaaa46ba618 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59172
I20260924 04:46:50.000475 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet bba38efbc1a544d686b61aaaa46ba618 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51956
I20260924 04:46:50.000504 2703 tablet_replica.cc:330] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529: stopping tablet replica
I20260924 04:46:50.000634 2703 raft_consensus.cc:2229] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:46:50.000748 2703 raft_consensus.cc:2258] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:50.000909 2705 tablet_replica.cc:330] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41: stopping tablet replica
I20260924 04:46:50.001086 2705 raft_consensus.cc:2229] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:46:50.001237 2705 raft_consensus.cc:2258] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:50.001224 2704 tablet_replica.cc:330] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3: stopping tablet replica
I20260924 04:46:50.001354 2704 raft_consensus.cc:2229] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:46:50.001494 2703 ts_tablet_manager.cc:1905] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:50.001590 2704 raft_consensus.cc:2258] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:50.001945 2705 ts_tablet_manager.cc:1905] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:50.002921 2704 ts_tablet_manager.cc:1905] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:50.004176 2703 ts_tablet_manager.cc:1918] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:46:50.004261 2703 log.cc:1196] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/bba38efbc1a544d686b61aaaa46ba618
I20260924 04:46:50.004503 2703 ts_tablet_manager.cc:1939] T bba38efbc1a544d686b61aaaa46ba618 P e53e00964bd145f08f7b05edab2fc529: Deleting consensus metadata
I20260924 04:46:50.004496 2705 ts_tablet_manager.cc:1918] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:46:50.004573 2705 log.cc:1196] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/bba38efbc1a544d686b61aaaa46ba618
I20260924 04:46:50.004920 2705 ts_tablet_manager.cc:1939] T bba38efbc1a544d686b61aaaa46ba618 P b2148181fd1542e7a514940edc036a41: Deleting consensus metadata
I20260924 04:46:50.005800 2110 catalog_manager.cc:4841] TS e53e00964bd145f08f7b05edab2fc529 (127.0.98.2:34831): tablet bba38efbc1a544d686b61aaaa46ba618 (table test-table [id=9daa79259d3b4736ab591ec1c167b46e]) successfully deleted
I20260924 04:46:50.005854 2704 ts_tablet_manager.cc:1918] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:46:50.005911 2704 log.cc:1196] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/wal/wals/bba38efbc1a544d686b61aaaa46ba618
I20260924 04:46:50.006539 2110 catalog_manager.cc:4841] TS b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863): tablet bba38efbc1a544d686b61aaaa46ba618 (table test-table [id=9daa79259d3b4736ab591ec1c167b46e]) successfully deleted
I20260924 04:46:50.006610 2704 ts_tablet_manager.cc:1939] T bba38efbc1a544d686b61aaaa46ba618 P 082d2b30eb4e424893f0e25b046c73f3: Deleting consensus metadata
I20260924 04:46:50.007198 2112 catalog_manager.cc:4841] TS 082d2b30eb4e424893f0e25b046c73f3 (127.0.98.1:40931): tablet bba38efbc1a544d686b61aaaa46ba618 (table test-table [id=9daa79259d3b4736ab591ec1c167b46e]) successfully deleted
Sep 24 04:46:50 dist-test-slave-hvcl krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225210, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:46:50 dist-test-slave-hvcl krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225210, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:50.034809 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:50.019521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37114 (local address 127.0.98.62:40401)
0924 04:46:50.019803 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:50.019810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:50.019825 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:46:50.019882 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:50.019884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:50.019927 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:50.020005 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:50.020701 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.021263 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.022072 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.022286 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.022378 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:50.031442 (+ 9064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:50.031446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:50.031448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:50.031483 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:50.033041 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:50.033534 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:50.033537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:50.033538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:50.033629 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:50.033926 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:50.033929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:50.033930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:50.034138 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:50.034244 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:50.034527 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 04:46:50.034636 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":116,"threads_started":1}
I20260924 04:46:50.037506 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:46:50.037899 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:46:50.042364 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8beeb350ff4e3baa5625f026aeab0a (DEFAULT_TABLE table=test-table [id=fc663b6d04f0455bb5d8611594c34efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:50.042335 2262 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8beeb350ff4e3baa5625f026aeab0a (DEFAULT_TABLE table=test-table [id=fc663b6d04f0455bb5d8611594c34efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:50.042400 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8beeb350ff4e3baa5625f026aeab0a (DEFAULT_TABLE table=test-table [id=fc663b6d04f0455bb5d8611594c34efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:50.042670 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a8beeb350ff4e3baa5625f026aeab0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:50.042670 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a8beeb350ff4e3baa5625f026aeab0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:50.042688 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a8beeb350ff4e3baa5625f026aeab0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:50.044699 2714 tablet_bootstrap.cc:492] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3: Bootstrap starting.
I20260924 04:46:50.045028 2716 tablet_bootstrap.cc:492] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41: Bootstrap starting.
I20260924 04:46:50.045598 2715 tablet_bootstrap.cc:492] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529: Bootstrap starting.
I20260924 04:46:50.045615 2714 tablet_bootstrap.cc:654] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:50.045887 2716 tablet_bootstrap.cc:654] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:50.046428 2715 tablet_bootstrap.cc:654] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:50.046782 2714 tablet_bootstrap.cc:492] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3: No bootstrap required, opened a new log
I20260924 04:46:50.046864 2714 ts_tablet_manager.cc:1397] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:46:50.047066 2716 tablet_bootstrap.cc:492] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41: No bootstrap required, opened a new log
I20260924 04:46:50.047155 2716 ts_tablet_manager.cc:1397] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:46:50.047457 2714 raft_consensus.cc:348] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.047588 2716 raft_consensus.cc:348] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.047677 2714 raft_consensus.cc:374] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:50.047731 2714 raft_consensus.cc:729] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082d2b30eb4e424893f0e25b046c73f3, State: Initialized, Role: FOLLOWER
I20260924 04:46:50.047772 2716 raft_consensus.cc:374] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:50.047825 2716 raft_consensus.cc:729] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2148181fd1542e7a514940edc036a41, State: Initialized, Role: FOLLOWER
I20260924 04:46:50.047886 2714 consensus_queue.cc:260] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.047959 2716 consensus_queue.cc:260] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.048256 2714 ts_tablet_manager.cc:1428] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 04:46:50.048354 2716 ts_tablet_manager.cc:1428] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:46:50.048712 2715 tablet_bootstrap.cc:492] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529: No bootstrap required, opened a new log
I20260924 04:46:50.048797 2715 ts_tablet_manager.cc:1397] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 04:46:50.049218 2715 raft_consensus.cc:348] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.049335 2715 raft_consensus.cc:374] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:50.049373 2715 raft_consensus.cc:729] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53e00964bd145f08f7b05edab2fc529, State: Initialized, Role: FOLLOWER
I20260924 04:46:50.049502 2715 consensus_queue.cc:260] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.049860 2715 ts_tablet_manager.cc:1428] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:46:50.150604 2697 raft_consensus.cc:482] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:46:50.150744 2697 raft_consensus.cc:504] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.151175 2697 leader_election.cc:288] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e53e00964bd145f08f7b05edab2fc529 (127.0.98.2:34831), b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863)
I20260924 04:46:50.151665 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8beeb350ff4e3baa5625f026aeab0a" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53e00964bd145f08f7b05edab2fc529" is_pre_election: true
I20260924 04:46:50.151681 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8beeb350ff4e3baa5625f026aeab0a" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2148181fd1542e7a514940edc036a41" is_pre_election: true
I20260924 04:46:50.151801 2422 raft_consensus.cc:2454] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 0.
I20260924 04:46:50.151810 2561 raft_consensus.cc:2454] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 0.
I20260924 04:46:50.152115 2214 leader_election.cc:302] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082d2b30eb4e424893f0e25b046c73f3, e53e00964bd145f08f7b05edab2fc529; no voters:
I20260924 04:46:50.152310 2697 raft_consensus.cc:2781] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:46:50.152377 2697 raft_consensus.cc:482] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:46:50.152407 2697 raft_consensus.cc:3037] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:50.153491 2697 raft_consensus.cc:504] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.153801 2697 leader_election.cc:288] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 election: Requested vote from peers e53e00964bd145f08f7b05edab2fc529 (127.0.98.2:34831), b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863)
I20260924 04:46:50.154139 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8beeb350ff4e3baa5625f026aeab0a" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e53e00964bd145f08f7b05edab2fc529"
I20260924 04:46:50.154209 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8beeb350ff4e3baa5625f026aeab0a" candidate_uuid: "082d2b30eb4e424893f0e25b046c73f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2148181fd1542e7a514940edc036a41"
I20260924 04:46:50.154256 2422 raft_consensus.cc:3037] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:50.154332 2561 raft_consensus.cc:3037] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:50.155227 2422 raft_consensus.cc:2454] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 1.
I20260924 04:46:50.155254 2561 raft_consensus.cc:2454] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 082d2b30eb4e424893f0e25b046c73f3 in term 1.
I20260924 04:46:50.155501 2214 leader_election.cc:302] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082d2b30eb4e424893f0e25b046c73f3, e53e00964bd145f08f7b05edab2fc529; no voters:
I20260924 04:46:50.155669 2697 raft_consensus.cc:2781] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:50.155753 2697 raft_consensus.cc:686] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [term 1 LEADER]: Becoming Leader. State: Replica: 082d2b30eb4e424893f0e25b046c73f3, State: Running, Role: LEADER
I20260924 04:46:50.155958 2697 consensus_queue.cc:237] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.157639 2120 catalog_manager.cc:5495] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 082d2b30eb4e424893f0e25b046c73f3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "082d2b30eb4e424893f0e25b046c73f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } health_report { overall_health: UNKNOWN } } }
I20260924 04:46:50.177435 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:50.174420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.0.98.1:40931)
0924 04:46:50.174681 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:50.174692 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:50.174704 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:46:50.174739 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:50.174741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:50.174779 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:50.174844 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:46:50.175114 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.175634 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.176474 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.176698 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.176778 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:50.177010 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:46:50.177082 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:46:50.177139 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:50.177236 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 04:46:50.177287 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":107,"threads_started":1}
I20260924 04:46:50.183418 2422 raft_consensus.cc:1261] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Refusing update from remote peer 082d2b30eb4e424893f0e25b046c73f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:50.183404 2561 raft_consensus.cc:1261] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Refusing update from remote peer 082d2b30eb4e424893f0e25b046c73f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:50.183799 2686 consensus_queue.cc:1035] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:50.183946 2637 consensus_queue.cc:1035] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:50.209378 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:46:50.209537 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:46:50.211473 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b: Sending DeleteTablet for 3 replicas of tablet 9a8beeb350ff4e3baa5625f026aeab0a
I20260924 04:46:50.212075 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a8beeb350ff4e3baa5625f026aeab0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33628
I20260924 04:46:50.212191 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a8beeb350ff4e3baa5625f026aeab0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51956
I20260924 04:46:50.212402 2705 tablet_replica.cc:330] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41: stopping tablet replica
I20260924 04:46:50.212416 2704 tablet_replica.cc:330] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3: stopping tablet replica
I20260924 04:46:50.212525 2705 raft_consensus.cc:2229] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:46:50.212532 2704 raft_consensus.cc:2229] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:46:50.212644 2705 raft_consensus.cc:2258] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:50.212720 2704 raft_consensus.cc:2258] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:50.212790 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:46:50.213088 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a8beeb350ff4e3baa5625f026aeab0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59172
W20260924 04:46:50.213183 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:46:50.213258 2703 tablet_replica.cc:330] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529: stopping tablet replica
I20260924 04:46:50.213361 2703 raft_consensus.cc:2229] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:46:50.213369 2704 ts_tablet_manager.cc:1905] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:50.213459 2703 raft_consensus.cc:2258] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:50.213594 2705 ts_tablet_manager.cc:1905] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:50.214277 2703 ts_tablet_manager.cc:1905] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:50.215852 2705 ts_tablet_manager.cc:1918] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:46:50.215929 2705 log.cc:1196] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/9a8beeb350ff4e3baa5625f026aeab0a
I20260924 04:46:50.215906 2704 ts_tablet_manager.cc:1918] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:46:50.215973 2704 log.cc:1196] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-0/wal/wals/9a8beeb350ff4e3baa5625f026aeab0a
I20260924 04:46:50.216176 2705 ts_tablet_manager.cc:1939] T 9a8beeb350ff4e3baa5625f026aeab0a P b2148181fd1542e7a514940edc036a41: Deleting consensus metadata
I20260924 04:46:50.216189 2704 ts_tablet_manager.cc:1939] T 9a8beeb350ff4e3baa5625f026aeab0a P 082d2b30eb4e424893f0e25b046c73f3: Deleting consensus metadata
I20260924 04:46:50.216223 2703 ts_tablet_manager.cc:1918] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:46:50.216275 2703 log.cc:1196] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/9a8beeb350ff4e3baa5625f026aeab0a
I20260924 04:46:50.216472 2703 ts_tablet_manager.cc:1939] T 9a8beeb350ff4e3baa5625f026aeab0a P e53e00964bd145f08f7b05edab2fc529: Deleting consensus metadata
I20260924 04:46:50.216985 2112 catalog_manager.cc:4841] TS 082d2b30eb4e424893f0e25b046c73f3 (127.0.98.1:40931): tablet 9a8beeb350ff4e3baa5625f026aeab0a (table test-table [id=fc663b6d04f0455bb5d8611594c34efe]) successfully deleted
I20260924 04:46:50.217036 2110 catalog_manager.cc:4841] TS e53e00964bd145f08f7b05edab2fc529 (127.0.98.2:34831): tablet 9a8beeb350ff4e3baa5625f026aeab0a (table test-table [id=fc663b6d04f0455bb5d8611594c34efe]) successfully deleted
I20260924 04:46:50.217300 2110 catalog_manager.cc:4841] TS b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863): tablet 9a8beeb350ff4e3baa5625f026aeab0a (table test-table [id=fc663b6d04f0455bb5d8611594c34efe]) successfully deleted
I20260924 04:46:50.219964 2261 tablet_service.cc:1467] Processing CreateTablet for tablet f20cb09576864dba853019257acdd1ab (DEFAULT_TABLE table=test-table [id=ec65f114abcb4b8bbc5764a29e8dcba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:50.220098 2542 tablet_service.cc:1467] Processing CreateTablet for tablet f20cb09576864dba853019257acdd1ab (DEFAULT_TABLE table=test-table [id=ec65f114abcb4b8bbc5764a29e8dcba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:50.220240 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20cb09576864dba853019257acdd1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:50.220204 2402 tablet_service.cc:1467] Processing CreateTablet for tablet f20cb09576864dba853019257acdd1ab (DEFAULT_TABLE table=test-table [id=ec65f114abcb4b8bbc5764a29e8dcba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:46:50.220297 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20cb09576864dba853019257acdd1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:50.220403 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20cb09576864dba853019257acdd1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:50.221899 2716 tablet_bootstrap.cc:492] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41: Bootstrap starting.
I20260924 04:46:50.222406 2714 tablet_bootstrap.cc:492] T f20cb09576864dba853019257acdd1ab P 082d2b30eb4e424893f0e25b046c73f3: Bootstrap starting.
I20260924 04:46:50.222440 2715 tablet_bootstrap.cc:492] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529: Bootstrap starting.
I20260924 04:46:50.222690 2716 tablet_bootstrap.cc:654] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:50.223181 2715 tablet_bootstrap.cc:654] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:50.223178 2714 tablet_bootstrap.cc:654] T f20cb09576864dba853019257acdd1ab P 082d2b30eb4e424893f0e25b046c73f3: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:50.223941 2716 tablet_bootstrap.cc:492] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41: No bootstrap required, opened a new log
I20260924 04:46:50.224035 2716 ts_tablet_manager.cc:1397] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:46:50.224164 2714 tablet_bootstrap.cc:492] T f20cb09576864dba853019257acdd1ab P 082d2b30eb4e424893f0e25b046c73f3: No bootstrap required, opened a new log
I20260924 04:46:50.224234 2714 ts_tablet_manager.cc:1397] T f20cb09576864dba853019257acdd1ab P 082d2b30eb4e424893f0e25b046c73f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:46:50.224450 2716 raft_consensus.cc:348] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.224551 2716 raft_consensus.cc:374] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:50.224582 2716 raft_consensus.cc:729] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2148181fd1542e7a514940edc036a41, State: Initialized, Role: FOLLOWER
I20260924 04:46:50.224582 2714 raft_consensus.cc:348] T f20cb09576864dba853019257acdd1ab P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.224666 2714 raft_consensus.cc:374] T f20cb09576864dba853019257acdd1ab P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:50.224694 2714 raft_consensus.cc:729] T f20cb09576864dba853019257acdd1ab P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082d2b30eb4e424893f0e25b046c73f3, State: Initialized, Role: FOLLOWER
I20260924 04:46:50.224702 2716 consensus_queue.cc:260] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.224812 2714 consensus_queue.cc:260] T f20cb09576864dba853019257acdd1ab P 082d2b30eb4e424893f0e25b046c73f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.224982 2716 ts_tablet_manager.cc:1428] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:46:50.225039 2714 ts_tablet_manager.cc:1428] T f20cb09576864dba853019257acdd1ab P 082d2b30eb4e424893f0e25b046c73f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:46:50.225679 2715 tablet_bootstrap.cc:492] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529: No bootstrap required, opened a new log
I20260924 04:46:50.225766 2715 ts_tablet_manager.cc:1397] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:46:50.226172 2715 raft_consensus.cc:348] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.226275 2715 raft_consensus.cc:374] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:50.226308 2715 raft_consensus.cc:729] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53e00964bd145f08f7b05edab2fc529, State: Initialized, Role: FOLLOWER
I20260924 04:46:50.226413 2715 consensus_queue.cc:260] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.226660 2715 ts_tablet_manager.cc:1428] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:46:50.266204 2721 raft_consensus.cc:482] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:46:50.266342 2721 raft_consensus.cc:504] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.267493 2721 leader_election.cc:288] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 082d2b30eb4e424893f0e25b046c73f3 (127.0.98.1:40931), b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863)
I20260924 04:46:50.271495 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:50.267848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57267 (local address 127.0.98.1:40931)
0924 04:46:50.267983 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:50.267988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:50.268422 (+ 434us) server_negotiation.cc:407] Connection header received
0924 04:46:50.268550 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:50.268553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:50.268605 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:50.268678 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:50.268924 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.269416 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.270516 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.270902 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.271013 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:50.271078 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:50.271309 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:46:50.271394 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:46:50.271507 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:50.267837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59999 (local address 127.0.98.3:32863)
0924 04:46:50.268353 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:50.268365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:50.268379 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:46:50.268431 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:50.268434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:50.268479 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:50.268559 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:46:50.268835 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.269462 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.270514 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.270937 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.271062 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:50.271137 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:50.271275 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 04:46:50.271350 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":70,"threads_started":1}
I20260924 04:46:50.271517 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:50.267742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32863 (local address 127.0.98.2:59999)
0924 04:46:50.268078 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:50.268090 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:50.268189 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:50.268577 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:50.268579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:50.268595 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:50.268723 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:50.268730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.269593 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.269595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:50.270403 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:50.270412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.271059 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.271063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:50.271174 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:50.271191 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:46:50.271245 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 04:46:50.271312 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":22,"thread_start_us":88,"threads_started":1}
I20260924 04:46:50.271522 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:50.267738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40931 (local address 127.0.98.2:57267)
0924 04:46:50.268325 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:50.268339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:50.268442 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:50.268702 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:50.268705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:50.268719 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:46:50.268826 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:50.268832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.269525 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.269529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:50.270394 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:50.270403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.271037 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.271041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:50.271149 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:50.271212 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 04:46:50.271283 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:46:50.271349 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":356,"thread_start_us":94,"threads_started":1}
I20260924 04:46:50.271960 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20cb09576864dba853019257acdd1ab" candidate_uuid: "e53e00964bd145f08f7b05edab2fc529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2148181fd1542e7a514940edc036a41" is_pre_election: true
I20260924 04:46:50.272122 2562 raft_consensus.cc:2454] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e53e00964bd145f08f7b05edab2fc529 in term 0.
I20260924 04:46:50.272107 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20cb09576864dba853019257acdd1ab" candidate_uuid: "e53e00964bd145f08f7b05edab2fc529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "082d2b30eb4e424893f0e25b046c73f3" is_pre_election: true
I20260924 04:46:50.272342 2282 raft_consensus.cc:2454] T f20cb09576864dba853019257acdd1ab P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e53e00964bd145f08f7b05edab2fc529 in term 0.
I20260924 04:46:50.272449 2354 leader_election.cc:302] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2148181fd1542e7a514940edc036a41, e53e00964bd145f08f7b05edab2fc529; no voters:
I20260924 04:46:50.272702 2721 raft_consensus.cc:2781] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:46:50.272761 2721 raft_consensus.cc:482] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:46:50.272802 2721 raft_consensus.cc:3037] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:50.273972 2721 raft_consensus.cc:504] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.274338 2721 leader_election.cc:288] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [CANDIDATE]: Term 1 election: Requested vote from peers 082d2b30eb4e424893f0e25b046c73f3 (127.0.98.1:40931), b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863)
I20260924 04:46:50.274616 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20cb09576864dba853019257acdd1ab" candidate_uuid: "e53e00964bd145f08f7b05edab2fc529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "082d2b30eb4e424893f0e25b046c73f3"
I20260924 04:46:50.274624 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20cb09576864dba853019257acdd1ab" candidate_uuid: "e53e00964bd145f08f7b05edab2fc529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2148181fd1542e7a514940edc036a41"
I20260924 04:46:50.274734 2562 raft_consensus.cc:3037] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:50.274734 2282 raft_consensus.cc:3037] T f20cb09576864dba853019257acdd1ab P 082d2b30eb4e424893f0e25b046c73f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:50.275774 2282 raft_consensus.cc:2454] T f20cb09576864dba853019257acdd1ab P 082d2b30eb4e424893f0e25b046c73f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e53e00964bd145f08f7b05edab2fc529 in term 1.
I20260924 04:46:50.275774 2562 raft_consensus.cc:2454] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e53e00964bd145f08f7b05edab2fc529 in term 1.
I20260924 04:46:50.276036 2354 leader_election.cc:302] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2148181fd1542e7a514940edc036a41, e53e00964bd145f08f7b05edab2fc529; no voters:
I20260924 04:46:50.276186 2721 raft_consensus.cc:2781] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:50.276376 2721 raft_consensus.cc:686] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [term 1 LEADER]: Becoming Leader. State: Replica: e53e00964bd145f08f7b05edab2fc529, State: Running, Role: LEADER
I20260924 04:46:50.276608 2721 consensus_queue.cc:237] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } }
I20260924 04:46:50.278659 2124 catalog_manager.cc:5495] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 reported cstate change: term changed from 0 to 1, leader changed from <none> to e53e00964bd145f08f7b05edab2fc529 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e53e00964bd145f08f7b05edab2fc529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e53e00964bd145f08f7b05edab2fc529" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 } health_report { overall_health: UNKNOWN } } }
I20260924 04:46:50.299038 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:50.295501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.0.98.2:34831)
0924 04:46:50.295785 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:50.295791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:50.295801 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:46:50.295877 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:50.295880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:50.295934 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:50.296044 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:46:50.296276 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.296804 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.297601 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.297836 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.297922 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:50.298158 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:46:50.298612 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:46:50.298710 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:50.298778 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 04:46:50.298832 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":118,"threads_started":1}
I20260924 04:46:50.309671 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:50.305957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40931 (local address 127.0.98.2:37197)
0924 04:46:50.306272 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:50.306285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:50.306375 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:50.306601 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:50.306603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:50.306610 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:46:50.307081 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:50.307089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.307873 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.307876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:50.308476 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:50.308485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.308921 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.308924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:50.309047 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:50.309093 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:46:50.309410 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:46:50.309413 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:46:50.309454 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 04:46:50.309530 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":97,"threads_started":1}
I20260924 04:46:50.309702 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:50.306079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37197 (local address 127.0.98.1:40931)
0924 04:46:50.306198 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:50.306202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:50.306364 (+ 162us) server_negotiation.cc:407] Connection header received
0924 04:46:50.306454 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:50.306456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:50.306511 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:50.306577 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:46:50.307226 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.307754 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.308579 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.308792 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.308878 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:50.309245 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:46:50.309339 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:46:50.309404 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:50.309523 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 04:46:50.309580 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:46:50.315405 2562 raft_consensus.cc:1261] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Refusing update from remote peer e53e00964bd145f08f7b05edab2fc529: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:50.315567 2282 raft_consensus.cc:1261] T f20cb09576864dba853019257acdd1ab P 082d2b30eb4e424893f0e25b046c73f3 [term 1 FOLLOWER]: Refusing update from remote peer e53e00964bd145f08f7b05edab2fc529: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:46:50.315860 2734 consensus_queue.cc:1035] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2148181fd1542e7a514940edc036a41" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:50.316030 2721 consensus_queue.cc:1035] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [LEADER]: Connected to new peer: Peer: permanent_uuid: "082d2b30eb4e424893f0e25b046c73f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:46:50.332618 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:50.329359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33227 (local address 127.0.98.2:34831)
0924 04:46:50.329614 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:50.329618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:50.329731 (+ 113us) server_negotiation.cc:407] Connection header received
0924 04:46:50.329848 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:50.329850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:50.329888 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:50.329944 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:46:50.330153 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.330639 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.331655 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.331974 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.332074 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:50.332288 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:46:50.332351 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:46:50.332401 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:50.332466 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 04:46:50.332507 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260924 04:46:50.332633 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:50.329310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34831 (local address 127.0.98.1:33227)
0924 04:46:50.329633 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:50.329658 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:50.329774 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:50.329976 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:50.329979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:50.329986 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:46:50.330075 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:50.330079 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.330710 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.330712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:50.331558 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:50.331567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:50.332077 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:50.332080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:50.332160 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:50.332194 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:46:50.332409 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:46:50.332413 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:46:50.332458 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 04:46:50.332521 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":121,"threads_started":1}
W20260924 04:46:50.333307 2688 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 04:46:51.891618 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:46:51.891777 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:46:51.894268 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c128ab0f054431d9b151ce18bc4352b: Sending DeleteTablet for 3 replicas of tablet f20cb09576864dba853019257acdd1ab
I20260924 04:46:51.894816 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet f20cb09576864dba853019257acdd1ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33628
I20260924 04:46:51.894955 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet f20cb09576864dba853019257acdd1ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59172
I20260924 04:46:51.895172 2769 tablet_replica.cc:330] T f20cb09576864dba853019257acdd1ab P 082d2b30eb4e424893f0e25b046c73f3: stopping tablet replica
I20260924 04:46:51.895239 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet f20cb09576864dba853019257acdd1ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:46:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51956
I20260924 04:46:51.895295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 2197
I20260924 04:46:51.896790 2770 tablet_replica.cc:330] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529: stopping tablet replica
I20260924 04:46:51.896901 2770 raft_consensus.cc:2229] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:46:51.897169 2770 raft_consensus.cc:2258] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:51.897392 2771 tablet_replica.cc:330] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41: stopping tablet replica
I20260924 04:46:51.897511 2771 raft_consensus.cc:2229] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:46:51.897624 2771 raft_consensus.cc:2258] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:46:51.897743 2770 ts_tablet_manager.cc:1905] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:51.898176 2771 ts_tablet_manager.cc:1905] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:46:51.900413 2771 ts_tablet_manager.cc:1918] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:46:51.900413 2770 ts_tablet_manager.cc:1918] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:46:51.900490 2771 log.cc:1196] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/f20cb09576864dba853019257acdd1ab
I20260924 04:46:51.900490 2770 log.cc:1196] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/f20cb09576864dba853019257acdd1ab
I20260924 04:46:51.900750 2771 ts_tablet_manager.cc:1939] T f20cb09576864dba853019257acdd1ab P b2148181fd1542e7a514940edc036a41: Deleting consensus metadata
I20260924 04:46:51.900768 2770 ts_tablet_manager.cc:1939] T f20cb09576864dba853019257acdd1ab P e53e00964bd145f08f7b05edab2fc529: Deleting consensus metadata
I20260924 04:46:51.901721 2110 catalog_manager.cc:4841] TS e53e00964bd145f08f7b05edab2fc529 (127.0.98.2:34831): tablet f20cb09576864dba853019257acdd1ab (table test-table [id=ec65f114abcb4b8bbc5764a29e8dcba1]) successfully deleted
W20260924 04:46:51.901934 2112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40931 (error 108)
I20260924 04:46:51.902172 2110 catalog_manager.cc:4841] TS b2148181fd1542e7a514940edc036a41 (127.0.98.3:32863): tablet f20cb09576864dba853019257acdd1ab (table test-table [id=ec65f114abcb4b8bbc5764a29e8dcba1]) successfully deleted
I20260924 04:46:51.902400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 2337
I20260924 04:46:51.903146 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:51.902557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40931 (local address 127.0.0.1:33664)
0924 04:46:51.902959 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:51.903051 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":72,"threads_started":1}
W20260924 04:46:51.903338 2112 catalog_manager.cc:4584] TS 082d2b30eb4e424893f0e25b046c73f3 (127.0.98.1:40931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f20cb09576864dba853019257acdd1ab: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40931: connect: Connection refused (error 111)
I20260924 04:46:51.908370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 2477
I20260924 04:46:51.912962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 2093
2026-09-24T04:46:51Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6060 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:46:51 dist-test-slave-hvcl krb5kdc[2776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:46:51 dist-test-slave-hvcl krb5kdc[2776](info): set up 2 sockets
Sep 24 04:46:51 dist-test-slave-hvcl krb5kdc[2776](info): commencing operation
krb5kdc: starting...
W20260924 04:46:52.870697 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:46:52 dist-test-slave-hvcl krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:46:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:46:52.993942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46685
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:53.104426 2789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:53.104681 2789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:53.104718 2789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:53.108618 2789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:46:53.108673 2789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:53.108691 2789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:53.108706 2789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:46:53.108719 2789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:46:53.114042 2789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:53.115127 2789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:53.116149 2789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:53.124825 2795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:53.124917 2789 server_base.cc:1034] running on GCE node
W20260924 04:46:53.124804 2794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:53.124817 2797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:53.125595 2789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:53.126763 2789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:53.127920 2789 hybrid_clock.cc:648] HybridClock initialized: now 1790225213127898 us; error 37 us; skew 500 ppm
Sep 24 04:46:53 dist-test-slave-hvcl krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:53.130714 2789 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:46:53.131634 2789 webserver.cc:460] Webserver started at http://127.0.98.62:35255/ using document root <none> and password file <none>
I20260924 04:46:53.132170 2789 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:53.132234 2789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:53.132426 2789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:53.134529 2789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "ee8818de3d98478eb927609843900721"
format_stamp: "Formatted at 2026-09-24 04:46:53 on dist-test-slave-hvcl"
server_key: "453464db494a15e96c599eb8c8a49c73"
server_key_iv: "1277a50eaeadd365a44032898a5b077c"
server_key_version: "encryptionkey@0"
I20260924 04:46:53.135114 2789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee8818de3d98478eb927609843900721"
format_stamp: "Formatted at 2026-09-24 04:46:53 on dist-test-slave-hvcl"
server_key: "453464db494a15e96c599eb8c8a49c73"
server_key_iv: "1277a50eaeadd365a44032898a5b077c"
server_key_version: "encryptionkey@0"
I20260924 04:46:53.138792 2789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:46:53.141248 2804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:53.142547 2789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:46:53.142704 2789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "ee8818de3d98478eb927609843900721"
format_stamp: "Formatted at 2026-09-24 04:46:53 on dist-test-slave-hvcl"
server_key: "453464db494a15e96c599eb8c8a49c73"
server_key_iv: "1277a50eaeadd365a44032898a5b077c"
server_key_version: "encryptionkey@0"
I20260924 04:46:53.142804 2789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:53.155494 2789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:53.158771 2789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:53.158949 2789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:53.165552 2789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46685
I20260924 04:46:53.165585 2856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46685 every 8 connection(s)
I20260924 04:46:53.166503 2789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:46:53.169341 2857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:53.170101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 2789
I20260924 04:46:53.170207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:46:53.170485 392 external_mini_cluster.cc:1428] Setting key 6f1e4ef163603fc34673b492e28eb659
I20260924 04:46:53.175524 2857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721: Bootstrap starting.
I20260924 04:46:53.178057 2857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:53.178951 2857 log.cc:824] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:46:53 dist-test-slave-hvcl krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:53.181002 2857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721: No bootstrap required, opened a new log
I20260924 04:46:53.184336 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:53.171930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.0.98.62:46685)
0924 04:46:53.172376 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:53.172391 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:53.172426 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:46:53.173108 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:53.173126 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:53.173457 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:53.173748 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:53.174622 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:53.175678 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:53.176817 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:53.177305 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:53.177425 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:53.180063 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:53.180067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:53.180080 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:53.180106 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:53.181669 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:53.182071 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:53.182075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:53.182081 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:53.182183 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:53.182428 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:53.182431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:53.182433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:53.182810 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:53.183025 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:53.183283 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 04:46:53.183580 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":163,"threads_started":1}
I20260924 04:46:53.184612 2857 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8818de3d98478eb927609843900721" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46685 } }
I20260924 04:46:53.184906 2857 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:53.184957 2857 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8818de3d98478eb927609843900721, State: Initialized, Role: FOLLOWER
I20260924 04:46:53.185407 2857 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8818de3d98478eb927609843900721" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46685 } }
I20260924 04:46:53.185573 2857 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:46:53.185637 2857 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:46:53.185737 2857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:53.186700 2857 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8818de3d98478eb927609843900721" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46685 } }
I20260924 04:46:53.187062 2857 leader_election.cc:302] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee8818de3d98478eb927609843900721; no voters:
I20260924 04:46:53.187330 2857 leader_election.cc:288] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:46:53.187433 2862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:53.187701 2862 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [term 1 LEADER]: Becoming Leader. State: Replica: ee8818de3d98478eb927609843900721, State: Running, Role: LEADER
I20260924 04:46:53.188022 2862 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8818de3d98478eb927609843900721" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46685 } }
I20260924 04:46:53.188494 2857 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:46:53.189514 2863 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee8818de3d98478eb927609843900721" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8818de3d98478eb927609843900721" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46685 } } }
I20260924 04:46:53.189641 2863 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:53.189900 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee8818de3d98478eb927609843900721. Latest consensus state: current_term: 1 leader_uuid: "ee8818de3d98478eb927609843900721" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8818de3d98478eb927609843900721" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46685 } } }
I20260924 04:46:53.189991 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721 [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:53.190032 2871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:46:53.193219 2871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:46:53.198906 2871 catalog_manager.cc:1347] Generated new cluster ID: 422b7935485444d4ba249d8d8b960084
I20260924 04:46:53.199007 2871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:46:53.222700 2871 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:46:53.224050 2871 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:46:53.229477 2871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee8818de3d98478eb927609843900721: Generated new TSK 0
I20260924 04:46:53.230302 2871 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:46:53.266829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46685
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:53.375627 2883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:53.375875 2883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:53.375916 2883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:53.379843 2883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:53.379897 2883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:53.379953 2883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:46:53.385515 2883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:53.386656 2883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:53.387720 2883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:53.397167 2891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:53.397328 2883 server_base.cc:1034] running on GCE node
W20260924 04:46:53.397167 2888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:53.397236 2889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:53.397909 2883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:53.398510 2883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:53.399718 2883 hybrid_clock.cc:648] HybridClock initialized: now 1790225213399699 us; error 37 us; skew 500 ppm
Sep 24 04:46:53 dist-test-slave-hvcl krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:53.402434 2883 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:46:53.403375 2883 webserver.cc:460] Webserver started at http://127.0.98.1:40959/ using document root <none> and password file <none>
I20260924 04:46:53.403909 2883 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:53.403973 2883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:53.404152 2883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:53.406160 2883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "c3bf0f17b0d64f088d526d8378ebeac3"
format_stamp: "Formatted at 2026-09-24 04:46:53 on dist-test-slave-hvcl"
server_key: "a468ea44a518987c70333b8df52851f3"
server_key_iv: "dd0fc42b53d467ac6df6ca2a54ff53da"
server_key_version: "encryptionkey@0"
I20260924 04:46:53.406700 2883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c3bf0f17b0d64f088d526d8378ebeac3"
format_stamp: "Formatted at 2026-09-24 04:46:53 on dist-test-slave-hvcl"
server_key: "a468ea44a518987c70333b8df52851f3"
server_key_iv: "dd0fc42b53d467ac6df6ca2a54ff53da"
server_key_version: "encryptionkey@0"
I20260924 04:46:53.410436 2883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:46:53.413072 2898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:53.414592 2883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:46:53.414723 2883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "c3bf0f17b0d64f088d526d8378ebeac3"
format_stamp: "Formatted at 2026-09-24 04:46:53 on dist-test-slave-hvcl"
server_key: "a468ea44a518987c70333b8df52851f3"
server_key_iv: "dd0fc42b53d467ac6df6ca2a54ff53da"
server_key_version: "encryptionkey@0"
I20260924 04:46:53.414835 2883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:53.439432 2883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:53.442706 2883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:53.442891 2883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:53.443533 2883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:53.444432 2883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:53.444482 2883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:53.444536 2883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:53.444566 2883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:53.452862 2883 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37519
I20260924 04:46:53.452975 3011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37519 every 8 connection(s)
I20260924 04:46:53.453787 2883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:46:53 dist-test-slave-hvcl krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:53.463231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 2883
I20260924 04:46:53.463352 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:46:53.463670 392 external_mini_cluster.cc:1428] Setting key 8e42c06e8f32b2565a1911a7df027bd9
I20260924 04:46:53.467633 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:53.455550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42873 (local address 127.0.98.62:46685)
0924 04:46:53.455669 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:53.455673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:53.456479 (+ 806us) server_negotiation.cc:407] Connection header received
0924 04:46:53.457864 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:53.457867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:53.457922 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:53.457994 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:53.459193 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:53.459716 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:53.460798 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:53.461021 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:53.461108 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:53.463842 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:53.463846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:53.463849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:53.463877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:53.465393 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:53.465988 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:53.465992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:53.465994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:53.466105 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:53.466405 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:53.466408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:53.466409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:53.466623 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:53.466716 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:53.467398 (+ 682us) server_negotiation.cc:299] Negotiation successful
0924 04:46:53.467501 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:46:53.468761 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:53.455878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46685 (local address 127.0.98.1:42873)
0924 04:46:53.456330 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:53.456372 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:53.457660 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:53.458177 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:53.458185 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:53.458546 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:53.459060 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:53.459076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:53.459825 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:53.459829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:53.460687 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:53.460695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:53.461150 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:53.461154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:53.461304 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:53.461836 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:53.461857 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:53.463681 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:53.465568 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:53.465573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:53.465585 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:53.465851 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:53.466189 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:53.466191 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:53.466193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:53.466330 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:53.466730 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:53.466737 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:53.467145 (+ 408us) client_negotiation.cc:769] Sending connection context
0924 04:46:53.467352 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 04:46:53.467731 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":128,"threads_started":1}
I20260924 04:46:53.470155 3012 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46685
I20260924 04:46:53.470432 3012 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:53.471055 3012 heartbeater.cc:511] Master 127.0.98.62:46685 requested a full tablet report, sending...
I20260924 04:46:53.472648 2818 ts_manager.cc:194] Registered new tserver with Master: c3bf0f17b0d64f088d526d8378ebeac3 (127.0.98.1:37519)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:46:53.473775 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:46:53.490666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46685
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:53.603096 3017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:53.603363 3017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:53.603405 3017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:53.607545 3017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:53.607609 3017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:53.607672 3017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:46:53.612924 3017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:53.614053 3017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:53.615129 3017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:53.624694 3023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:53.624672 3022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:53.624658 3025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:53.624776 3017 server_base.cc:1034] running on GCE node
I20260924 04:46:53.625304 3017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:53.625939 3017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:53.627110 3017 hybrid_clock.cc:648] HybridClock initialized: now 1790225213627094 us; error 32 us; skew 500 ppm
Sep 24 04:46:53 dist-test-slave-hvcl krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:53.629715 3017 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:46:53.630609 3017 webserver.cc:460] Webserver started at http://127.0.98.2:43205/ using document root <none> and password file <none>
I20260924 04:46:53.631187 3017 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:53.631247 3017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:53.631433 3017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:53.633427 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "14cf4d3df9a445fd9eb9ed0109ee9d28"
format_stamp: "Formatted at 2026-09-24 04:46:53 on dist-test-slave-hvcl"
server_key: "b2a570c0725fb020d61b6ee9c10dd3da"
server_key_iv: "b54d024ba3e65f44f7d802b2b09ea938"
server_key_version: "encryptionkey@0"
I20260924 04:46:53.633944 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "14cf4d3df9a445fd9eb9ed0109ee9d28"
format_stamp: "Formatted at 2026-09-24 04:46:53 on dist-test-slave-hvcl"
server_key: "b2a570c0725fb020d61b6ee9c10dd3da"
server_key_iv: "b54d024ba3e65f44f7d802b2b09ea938"
server_key_version: "encryptionkey@0"
I20260924 04:46:53.637660 3017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:46:53.640106 3032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:53.641453 3017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:46:53.641582 3017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "14cf4d3df9a445fd9eb9ed0109ee9d28"
format_stamp: "Formatted at 2026-09-24 04:46:53 on dist-test-slave-hvcl"
server_key: "b2a570c0725fb020d61b6ee9c10dd3da"
server_key_iv: "b54d024ba3e65f44f7d802b2b09ea938"
server_key_version: "encryptionkey@0"
I20260924 04:46:53.641688 3017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:53.660327 3017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:53.663654 3017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:53.663861 3017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:53.664470 3017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:53.665294 3017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:53.665349 3017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:53.665402 3017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:53.665427 3017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:53.676020 3017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34239
I20260924 04:46:53.676062 3145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34239 every 8 connection(s)
I20260924 04:46:53.677197 3017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 04:46:53 dist-test-slave-hvcl krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:53.687072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 3017
I20260924 04:46:53.687171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:46:53.687410 392 external_mini_cluster.cc:1428] Setting key 988f5aea58759a0afc3144c3eb27f9f0
I20260924 04:46:53.690279 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:53.679019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54433 (local address 127.0.98.62:46685)
0924 04:46:53.679149 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:53.679153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:53.680023 (+ 870us) server_negotiation.cc:407] Connection header received
0924 04:46:53.681317 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:53.681320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:53.681364 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:53.681432 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:53.682625 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:53.683185 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:53.684279 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:53.684492 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:53.684563 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:53.686793 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:53.686797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:53.686800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:53.686826 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:53.688233 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:53.688702 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:53.688705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:53.688708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:53.688782 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:53.689116 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:53.689120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:53.689121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:53.689366 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:53.689458 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:53.689968 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 04:46:53.690124 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:46:53.691105 3148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:53.679424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46685 (local address 127.0.98.2:54433)
0924 04:46:53.679881 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:53.679923 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:53.681141 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:53.681595 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:53.681602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:53.681946 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:53.682483 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:53.682497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:53.683288 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:53.683291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:53.684173 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:53.684182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:53.684598 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:53.684600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:53.684730 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:53.685271 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:53.685287 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:53.686650 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:53.688329 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:53.688335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:53.688349 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:53.688587 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:53.688857 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:53.688860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:53.688862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:53.688992 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:53.689476 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:53.689481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:53.689722 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 04:46:53.689877 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 04:46:53.690200 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":117,"threads_started":1}
I20260924 04:46:53.692135 3146 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46685
I20260924 04:46:53.692355 3146 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:53.692848 3146 heartbeater.cc:511] Master 127.0.98.62:46685 requested a full tablet report, sending...
I20260924 04:46:53.693814 2818 ts_manager.cc:194] Registered new tserver with Master: 14cf4d3df9a445fd9eb9ed0109ee9d28 (127.0.98.2:34239)
I20260924 04:46:53.694375 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54433
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:46:53.714671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46685
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:53.822502 3151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:53.822752 3151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:53.822813 3151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:53.826921 3151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:53.827008 3151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:53.827083 3151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:46:53.832367 3151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:53.833406 3151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:53.834421 3151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:53.843703 3157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:53.843703 3156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:53.843734 3159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:53.844035 3151 server_base.cc:1034] running on GCE node
I20260924 04:46:53.844404 3151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:53.845060 3151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:53.846207 3151 hybrid_clock.cc:648] HybridClock initialized: now 1790225213846179 us; error 45 us; skew 500 ppm
Sep 24 04:46:53 dist-test-slave-hvcl krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:53.848866 3151 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:46:53.849710 3151 webserver.cc:460] Webserver started at http://127.0.98.3:38661/ using document root <none> and password file <none>
I20260924 04:46:53.850229 3151 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:53.850291 3151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:53.850477 3151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:53.852526 3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "2813147459e443d8972bbaf3c2a2a000"
format_stamp: "Formatted at 2026-09-24 04:46:53 on dist-test-slave-hvcl"
server_key: "31234a0bf7246793c2cd0778efadba0a"
server_key_iv: "d62f47903e03be92187a5c2a0853e67c"
server_key_version: "encryptionkey@0"
I20260924 04:46:53.853042 3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2813147459e443d8972bbaf3c2a2a000"
format_stamp: "Formatted at 2026-09-24 04:46:53 on dist-test-slave-hvcl"
server_key: "31234a0bf7246793c2cd0778efadba0a"
server_key_iv: "d62f47903e03be92187a5c2a0853e67c"
server_key_version: "encryptionkey@0"
I20260924 04:46:53.856583 3151 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:46:53.858888 3166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:53.860112 3151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:46:53.860212 3151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "2813147459e443d8972bbaf3c2a2a000"
format_stamp: "Formatted at 2026-09-24 04:46:53 on dist-test-slave-hvcl"
server_key: "31234a0bf7246793c2cd0778efadba0a"
server_key_iv: "d62f47903e03be92187a5c2a0853e67c"
server_key_version: "encryptionkey@0"
I20260924 04:46:53.860302 3151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:53.869895 3151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:53.872854 3151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:53.873020 3151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:53.873557 3151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:53.874336 3151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:53.874382 3151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:53.874432 3151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:53.874461 3151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:53.882665 3151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35165
I20260924 04:46:53.882705 3279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35165 every 8 connection(s)
I20260924 04:46:53.883553 3151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:46:53.891456 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 3151
I20260924 04:46:53.891562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:46:53.891958 392 external_mini_cluster.cc:1428] Setting key 1b096021dd0e4db9e8e72d52c5879020
Sep 24 04:46:53 dist-test-slave-hvcl krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:53.897207 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:53.885290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33195 (local address 127.0.98.62:46685)
0924 04:46:53.885414 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:53.885418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:53.886228 (+ 810us) server_negotiation.cc:407] Connection header received
0924 04:46:53.887630 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:53.887634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:53.887685 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:53.887756 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:53.888927 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:53.889523 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:53.890714 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:53.890924 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:53.891039 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:53.893851 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:53.893854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:53.893856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:53.893880 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:53.895325 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:53.895813 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:53.895816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:53.895817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:53.895888 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:53.896183 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:53.896186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:53.896188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:53.896359 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:53.896442 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:53.896925 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 04:46:53.897042 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:46:53.898016 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:53.885612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46685 (local address 127.0.98.3:33195)
0924 04:46:53.886068 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:53.886109 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:53.887364 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:53.887931 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:53.887937 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:53.888289 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:53.888790 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:53.888810 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:53.889634 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:53.889636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:53.890601 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:53.890611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:53.891082 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:53.891086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:53.891258 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:53.891835 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:53.891851 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:53.893680 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:53.895431 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:53.895437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:53.895453 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:53.895722 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:53.895965 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:53.895968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:53.895970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:53.896101 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:53.896452 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:53.896459 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:53.896708 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 04:46:53.896874 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 04:46:53.897195 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
I20260924 04:46:53.899055 3280 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46685
I20260924 04:46:53.899293 3280 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:53.899819 3280 heartbeater.cc:511] Master 127.0.98.62:46685 requested a full tablet report, sending...
I20260924 04:46:53.900856 2818 ts_manager.cc:194] Registered new tserver with Master: 2813147459e443d8972bbaf3c2a2a000 (127.0.98.3:35165)
I20260924 04:46:53.901440 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33195
I20260924 04:46:53.905932 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:46:53.918074 2860 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:46:53.916608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.0.98.62:46685)
0924 04:46:53.916787 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:53.916792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:53.916906 (+ 114us) server_negotiation.cc:407] Connection header received
0924 04:46:53.917041 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:53.917044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:53.917091 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:53.917156 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:53.917940 (+ 784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42036: BlockingRecv error: recv got EOF from 127.0.0.1:42036 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 04:46:53.919368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 2883
I20260924 04:46:53.924222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 3017
I20260924 04:46:53.928614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 3151
I20260924 04:46:53.933116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 2789
2026-09-24T04:46:53Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2018 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:46:53 dist-test-slave-hvcl krb5kdc[3292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:46:53 dist-test-slave-hvcl krb5kdc[3292](info): set up 2 sockets
Sep 24 04:46:53 dist-test-slave-hvcl krb5kdc[3292](info): commencing operation
krb5kdc: starting...
W20260924 04:46:54.890697 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:46:54 dist-test-slave-hvcl krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:46:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:46:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:46:55.010139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37507
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:55.118485 3305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:55.118818 3305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:55.118863 3305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:55.122944 3305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:46:55.123030 3305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:55.123049 3305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:55.123065 3305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:46:55.123078 3305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:46:55.128350 3305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:55.129349 3305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:55.130319 3305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:46:55.138595 3305 server_base.cc:1034] running on GCE node
W20260924 04:46:55.138691 3310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:55.138706 3311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:55.138937 3313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:55.139497 3305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:55.140631 3305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:55.141849 3305 hybrid_clock.cc:648] HybridClock initialized: now 1790225215141819 us; error 45 us; skew 500 ppm
Sep 24 04:46:55 dist-test-slave-hvcl krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:55.148100 3305 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:46:55.148986 3305 webserver.cc:460] Webserver started at http://127.0.98.62:41391/ using document root <none> and password file <none>
I20260924 04:46:55.149516 3305 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:55.149575 3305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:55.149756 3305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:55.151847 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "427ac54009ef4b65847821302ec8baa5"
format_stamp: "Formatted at 2026-09-24 04:46:55 on dist-test-slave-hvcl"
server_key: "5f0fe8b6983fc51101948cb0af8cf379"
server_key_iv: "06002d1450ffaec17f495fb9cc35a0b2"
server_key_version: "encryptionkey@0"
I20260924 04:46:55.152395 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "427ac54009ef4b65847821302ec8baa5"
format_stamp: "Formatted at 2026-09-24 04:46:55 on dist-test-slave-hvcl"
server_key: "5f0fe8b6983fc51101948cb0af8cf379"
server_key_iv: "06002d1450ffaec17f495fb9cc35a0b2"
server_key_version: "encryptionkey@0"
I20260924 04:46:55.156148 3305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 04:46:55.158488 3320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:55.159819 3305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:46:55.159969 3305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "427ac54009ef4b65847821302ec8baa5"
format_stamp: "Formatted at 2026-09-24 04:46:55 on dist-test-slave-hvcl"
server_key: "5f0fe8b6983fc51101948cb0af8cf379"
server_key_iv: "06002d1450ffaec17f495fb9cc35a0b2"
server_key_version: "encryptionkey@0"
I20260924 04:46:55.160073 3305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:55.184453 3305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:55.187439 3305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:55.187614 3305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:55.194116 3305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37507
I20260924 04:46:55.194393 3372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37507 every 8 connection(s)
I20260924 04:46:55.195044 3305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:46:55.196187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 3305
I20260924 04:46:55.196311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:46:55.196605 392 external_mini_cluster.cc:1428] Setting key 7525c29cb215ef3b2bbea69a85a6d953
I20260924 04:46:55.198304 3373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:55.205024 3373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5: Bootstrap starting.
Sep 24 04:46:55 dist-test-slave-hvcl krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:55.207679 3373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:55.208544 3373 log.cc:824] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:55.210822 3373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5: No bootstrap required, opened a new log
I20260924 04:46:55.211133 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:55.197818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.0.98.62:37507)
0924 04:46:55.198229 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:55.198256 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:55.198296 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:46:55.199166 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:55.199189 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:55.199612 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:55.199974 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:55.200629 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:55.202083 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:55.202934 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:55.203660 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:55.203830 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:55.206136 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:55.206143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:55.206160 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:55.206194 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:55.208135 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:55.208568 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:55.208574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:55.208580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:55.208709 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:55.209010 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:55.209012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:55.209014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:55.209439 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:55.209608 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:55.209878 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 04:46:55.210159 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":121,"threads_started":1}
I20260924 04:46:55.213901 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:46:55.214562 3373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427ac54009ef4b65847821302ec8baa5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } }
I20260924 04:46:55.214852 3373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:55.214918 3373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427ac54009ef4b65847821302ec8baa5, State: Initialized, Role: FOLLOWER
I20260924 04:46:55.215401 3373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427ac54009ef4b65847821302ec8baa5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } }
I20260924 04:46:55.215543 3373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:46:55.215601 3373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:46:55.215687 3373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:55.216619 3373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427ac54009ef4b65847821302ec8baa5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } }
I20260924 04:46:55.216940 3373 leader_election.cc:302] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 427ac54009ef4b65847821302ec8baa5; no voters:
I20260924 04:46:55.217209 3373 leader_election.cc:288] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:46:55.217357 3379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:55.217631 3379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [term 1 LEADER]: Becoming Leader. State: Replica: 427ac54009ef4b65847821302ec8baa5, State: Running, Role: LEADER
I20260924 04:46:55.218005 3379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427ac54009ef4b65847821302ec8baa5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } }
I20260924 04:46:55.218359 3373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:46:55.219753 3381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 427ac54009ef4b65847821302ec8baa5. Latest consensus state: current_term: 1 leader_uuid: "427ac54009ef4b65847821302ec8baa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427ac54009ef4b65847821302ec8baa5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } }
I20260924 04:46:55.219889 3381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:55.220129 3380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "427ac54009ef4b65847821302ec8baa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427ac54009ef4b65847821302ec8baa5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } }
I20260924 04:46:55.220212 3380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5 [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:55.220643 3392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:46:55.223778 3392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:46:55.229867 3392 catalog_manager.cc:1347] Generated new cluster ID: f99ec11f45ce46e2a116d6124f439ab0
I20260924 04:46:55.229938 3392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:46:55.233444 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:55.227631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.0.98.62:37507)
0924 04:46:55.227748 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:55.227753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:55.228314 (+ 561us) server_negotiation.cc:407] Connection header received
0924 04:46:55.228476 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:55.228479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:55.228531 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:55.228631 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:55.229236 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:55.230038 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:55.231183 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:55.231557 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:55.231664 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:55.232507 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:55.232511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:55.232514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:46:55.233029 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:55.233063 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:46:55.233081 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:55.233169 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:55.233235 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 04:46:55.233305 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:46:55.253158 3392 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:46:55.254451 3392 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:46:55.266564 3392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 427ac54009ef4b65847821302ec8baa5: Generated new TSK 0
I20260924 04:46:55.267508 3392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:46:55.274775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 3305
2026-09-24T04:46:55Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1340 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:46:55 dist-test-slave-hvcl krb5kdc[3406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:46:55 dist-test-slave-hvcl krb5kdc[3406](info): set up 2 sockets
Sep 24 04:46:55 dist-test-slave-hvcl krb5kdc[3406](info): commencing operation
krb5kdc: starting...
W20260924 04:46:56.226648 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:46:56 dist-test-slave-hvcl krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:46:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:46:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:46:56.355432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34795
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:56.465117 3419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:56.465391 3419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:56.465431 3419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:56.469563 3419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:46:56.469645 3419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:56.469664 3419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:56.469681 3419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:46:56.469694 3419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:46:56.475095 3419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:56.476186 3419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:56.477205 3419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:56.485707 3424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:56.485700 3427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:56.485700 3425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:56.485747 3419 server_base.cc:1034] running on GCE node
I20260924 04:46:56.486429 3419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:56.487560 3419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:56.488734 3419 hybrid_clock.cc:648] HybridClock initialized: now 1790225216488677 us; error 71 us; skew 500 ppm
Sep 24 04:46:56 dist-test-slave-hvcl krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:56.491464 3419 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:46:56.492363 3419 webserver.cc:460] Webserver started at http://127.0.98.62:35869/ using document root <none> and password file <none>
I20260924 04:46:56.492887 3419 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:56.492944 3419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:56.493145 3419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:56.495164 3419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "0b1633dd3cf949a6a821258eaca6f1b9"
format_stamp: "Formatted at 2026-09-24 04:46:56 on dist-test-slave-hvcl"
server_key: "a88c74286c040ddc695e3d0da7055558"
server_key_iv: "7423ec36a05134dd1c313d1b76df0df3"
server_key_version: "encryptionkey@0"
I20260924 04:46:56.495689 3419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b1633dd3cf949a6a821258eaca6f1b9"
format_stamp: "Formatted at 2026-09-24 04:46:56 on dist-test-slave-hvcl"
server_key: "a88c74286c040ddc695e3d0da7055558"
server_key_iv: "7423ec36a05134dd1c313d1b76df0df3"
server_key_version: "encryptionkey@0"
I20260924 04:46:56.499446 3419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:46:56.501832 3434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:56.503238 3419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:46:56.503360 3419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "0b1633dd3cf949a6a821258eaca6f1b9"
format_stamp: "Formatted at 2026-09-24 04:46:56 on dist-test-slave-hvcl"
server_key: "a88c74286c040ddc695e3d0da7055558"
server_key_iv: "7423ec36a05134dd1c313d1b76df0df3"
server_key_version: "encryptionkey@0"
I20260924 04:46:56.503459 3419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:56.519954 3419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:56.523180 3419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:56.523357 3419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:56.530030 3419 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34795
I20260924 04:46:56.530333 3486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34795 every 8 connection(s)
I20260924 04:46:56.531318 3419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:46:56.534430 3487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:56.541536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 3419
I20260924 04:46:56.541668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:46:56.541898 3487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9: Bootstrap starting.
I20260924 04:46:56.541953 392 external_mini_cluster.cc:1428] Setting key 82a65e02462e27f6437417278d2f7f72
I20260924 04:46:56.544780 3487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:56.545635 3487 log.cc:824] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:56.547750 3487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9: No bootstrap required, opened a new log
I20260924 04:46:56.551834 3487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1633dd3cf949a6a821258eaca6f1b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34795 } }
I20260924 04:46:56.552112 3487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:56.552166 3487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b1633dd3cf949a6a821258eaca6f1b9, State: Initialized, Role: FOLLOWER
I20260924 04:46:56.552775 3487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1633dd3cf949a6a821258eaca6f1b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34795 } }
I20260924 04:46:56.552964 3487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:46:56.553038 3487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:46:56.553138 3487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:56.554390 3487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1633dd3cf949a6a821258eaca6f1b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34795 } }
Sep 24 04:46:56 dist-test-slave-hvcl krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:56.554786 3487 leader_election.cc:302] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b1633dd3cf949a6a821258eaca6f1b9; no voters:
I20260924 04:46:56.555171 3487 leader_election.cc:288] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:46:56.555255 3492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:56.555455 3492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [term 1 LEADER]: Becoming Leader. State: Replica: 0b1633dd3cf949a6a821258eaca6f1b9, State: Running, Role: LEADER
I20260924 04:46:56.555773 3492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1633dd3cf949a6a821258eaca6f1b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34795 } }
I20260924 04:46:56.556624 3487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:46:56.557228 3494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b1633dd3cf949a6a821258eaca6f1b9. Latest consensus state: current_term: 1 leader_uuid: "0b1633dd3cf949a6a821258eaca6f1b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1633dd3cf949a6a821258eaca6f1b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34795 } } }
I20260924 04:46:56.557348 3494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:56.557634 3493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b1633dd3cf949a6a821258eaca6f1b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1633dd3cf949a6a821258eaca6f1b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34795 } } }
I20260924 04:46:56.557709 3493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:56.561651 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:56.543719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.0.98.62:34795)
0924 04:46:56.546462 (+ 2743us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:56.546478 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:56.546513 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:46:56.547349 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:56.547374 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:56.547852 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:56.548166 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:56.549021 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:56.549983 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:56.551194 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:56.551917 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:56.552094 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:56.555218 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:56.555225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:56.555244 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:56.555277 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:56.557296 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:56.558205 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:56.558210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:56.558216 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:56.558361 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:56.558748 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:56.558752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:56.558754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:56.559312 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:56.559518 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:56.560257 (+ 739us) server_negotiation.cc:299] Negotiation successful
0924 04:46:56.560570 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2581,"thread_start_us":119,"threads_started":1}
W20260924 04:46:56.565716 3507 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:46:56.565793 3507 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:46:56.565848 3498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:46:56.566550 3498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:46:56.572176 3498 catalog_manager.cc:1347] Generated new cluster ID: fb611b2be4fc4239917002f9332b0105
I20260924 04:46:56.572233 3498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:46:56.598654 3498 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:46:56.599646 3498 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:46:56.605587 3498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b1633dd3cf949a6a821258eaca6f1b9: Generated new TSK 0
I20260924 04:46:56.606280 3498 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:46:56.637693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34795
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:56.743459 3513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:56.743716 3513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:56.743762 3513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:56.747722 3513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:56.747778 3513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:56.747839 3513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:46:56.753013 3513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:56.754005 3513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:56.755017 3513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:56.764421 3519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:56.764480 3521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:56.764488 3513 server_base.cc:1034] running on GCE node
W20260924 04:46:56.764426 3518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:56.765031 3513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:56.765650 3513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:56.766808 3513 hybrid_clock.cc:648] HybridClock initialized: now 1790225216766783 us; error 38 us; skew 500 ppm
Sep 24 04:46:56 dist-test-slave-hvcl krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:56.769500 3513 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:46:56.770386 3513 webserver.cc:460] Webserver started at http://127.0.98.1:36387/ using document root <none> and password file <none>
I20260924 04:46:56.770905 3513 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:56.770987 3513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:56.771190 3513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:56.773419 3513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "0a673cee688942e6b052f748c84604be"
format_stamp: "Formatted at 2026-09-24 04:46:56 on dist-test-slave-hvcl"
server_key: "d0dff87ca90b4e9aa3ef836f9aa7aaa6"
server_key_iv: "762075da7731fbca9a3f52defcfbcca8"
server_key_version: "encryptionkey@0"
I20260924 04:46:56.773939 3513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0a673cee688942e6b052f748c84604be"
format_stamp: "Formatted at 2026-09-24 04:46:56 on dist-test-slave-hvcl"
server_key: "d0dff87ca90b4e9aa3ef836f9aa7aaa6"
server_key_iv: "762075da7731fbca9a3f52defcfbcca8"
server_key_version: "encryptionkey@0"
I20260924 04:46:56.777480 3513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:46:56.779858 3528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:56.781106 3513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 04:46:56.781219 3513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "0a673cee688942e6b052f748c84604be"
format_stamp: "Formatted at 2026-09-24 04:46:56 on dist-test-slave-hvcl"
server_key: "d0dff87ca90b4e9aa3ef836f9aa7aaa6"
server_key_iv: "762075da7731fbca9a3f52defcfbcca8"
server_key_version: "encryptionkey@0"
I20260924 04:46:56.781315 3513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:56.796401 3513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:56.799520 3513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:56.799696 3513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:56.800280 3513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:56.801081 3513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:56.801127 3513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:56.801180 3513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:56.801214 3513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:56.810075 3513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37769
I20260924 04:46:56.810150 3641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37769 every 8 connection(s)
I20260924 04:46:56.811021 3513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:46:56.813824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 3513
I20260924 04:46:56.813939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:46:56.814236 392 external_mini_cluster.cc:1428] Setting key faf5d256832164b089c5a945b08d808c
Sep 24 04:46:56 dist-test-slave-hvcl krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:46:56.824997 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:56.812816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60631 (local address 127.0.98.62:34795)
0924 04:46:56.813039 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:56.813043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:56.813842 (+ 799us) server_negotiation.cc:407] Connection header received
0924 04:46:56.815402 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:56.815405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:56.815454 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:56.815524 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:56.816627 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:56.817159 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:56.818505 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:56.818744 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:56.818827 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:56.821642 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:56.821645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:56.821647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:56.821672 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:56.823049 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:56.823517 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:56.823520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:56.823521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:56.823618 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:56.823921 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:56.823924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:56.823925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:56.824113 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:56.824189 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:56.824703 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 04:46:56.824836 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:46:56.825930 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:56.813203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34795 (local address 127.0.98.1:60631)
0924 04:46:56.813666 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:56.813723 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:56.815211 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:56.815685 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:56.815692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:56.816026 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:56.816499 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:56.816512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:56.817276 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:56.817279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:56.818386 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:56.818396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:56.818884 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:56.818889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:56.819120 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:56.819707 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:56.819727 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:56.821455 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:56.823143 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:56.823148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:56.823160 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:56.823437 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:56.823717 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:56.823720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:56.823722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:56.823852 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:56.824185 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:56.824190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:56.824446 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 04:46:56.824670 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 04:46:56.824989 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260924 04:46:56.827266 3642 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34795
I20260924 04:46:56.827497 3642 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:56.827971 3642 heartbeater.cc:511] Master 127.0.98.62:34795 requested a full tablet report, sending...
I20260924 04:46:56.829459 3451 ts_manager.cc:194] Registered new tserver with Master: 0a673cee688942e6b052f748c84604be (127.0.98.1:37769)
I20260924 04:46:56.830607 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:46:56.842782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34795
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:56.949743 3647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:56.950001 3647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:56.950063 3647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:56.954129 3647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:56.954201 3647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:56.954285 3647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:46:56.959728 3647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:56.960834 3647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:56.961870 3647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:56.971665 3653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:56.971673 3655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:56.971699 3652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:56.971990 3647 server_base.cc:1034] running on GCE node
I20260924 04:46:56.972398 3647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:56.973070 3647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:56.974258 3647 hybrid_clock.cc:648] HybridClock initialized: now 1790225216974220 us; error 50 us; skew 500 ppm
Sep 24 04:46:56 dist-test-slave-hvcl krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:56.977200 3647 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:46:56.978235 3647 webserver.cc:460] Webserver started at http://127.0.98.2:46863/ using document root <none> and password file <none>
I20260924 04:46:56.978791 3647 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:56.978865 3647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:56.979118 3647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:56.981176 3647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "d2035222dc674a04917693e5dd4ddd85"
format_stamp: "Formatted at 2026-09-24 04:46:56 on dist-test-slave-hvcl"
server_key: "414b864d95fe25c369abbce84d15577c"
server_key_iv: "73818186d8a29d584205febb9dd4ea78"
server_key_version: "encryptionkey@0"
I20260924 04:46:56.981729 3647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d2035222dc674a04917693e5dd4ddd85"
format_stamp: "Formatted at 2026-09-24 04:46:56 on dist-test-slave-hvcl"
server_key: "414b864d95fe25c369abbce84d15577c"
server_key_iv: "73818186d8a29d584205febb9dd4ea78"
server_key_version: "encryptionkey@0"
I20260924 04:46:56.985473 3647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 04:46:56.987980 3662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:56.989296 3647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:46:56.989409 3647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "d2035222dc674a04917693e5dd4ddd85"
format_stamp: "Formatted at 2026-09-24 04:46:56 on dist-test-slave-hvcl"
server_key: "414b864d95fe25c369abbce84d15577c"
server_key_iv: "73818186d8a29d584205febb9dd4ea78"
server_key_version: "encryptionkey@0"
I20260924 04:46:56.989516 3647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:57.001864 3647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:57.005029 3647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:57.005204 3647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:57.005786 3647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:57.006592 3647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:57.006661 3647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:57.006714 3647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:57.006742 3647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:57.015378 3647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33773
I20260924 04:46:57.015425 3775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33773 every 8 connection(s)
I20260924 04:46:57.016232 3647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:46:57.018646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 3647
I20260924 04:46:57.018774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:46:57.019086 392 external_mini_cluster.cc:1428] Setting key 6b61ac67bfd40fe9438196c2673f7d56
Sep 24 04:46:57 dist-test-slave-hvcl krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:46:57.030246 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:57.017954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57043 (local address 127.0.98.62:34795)
0924 04:46:57.018123 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:57.018126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:57.019018 (+ 892us) server_negotiation.cc:407] Connection header received
0924 04:46:57.020644 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:57.020648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:57.020696 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:57.020767 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:57.021932 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:57.022444 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:57.023615 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:57.023832 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:57.023917 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:57.026643 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:57.026646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:57.026648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:57.026672 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:57.028160 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:57.028689 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:57.028693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:57.028695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:57.028791 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:57.029157 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:57.029160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:57.029161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:57.029355 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:57.029433 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:57.030005 (+ 572us) server_negotiation.cc:299] Negotiation successful
0924 04:46:57.030114 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:46:57.031220 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:57.018300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34795 (local address 127.0.98.2:57043)
0924 04:46:57.018861 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:57.018905 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:57.020433 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:57.020951 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:57.020959 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:57.021303 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:57.021802 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:57.021814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:57.022566 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:57.022569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:57.023490 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:57.023499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:57.023964 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:57.023968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:57.024133 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:57.024749 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:57.024766 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:57.026409 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:57.028279 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:57.028284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:57.028297 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:57.028553 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:57.028907 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:57.028909 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:57.028911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:57.029064 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:57.029440 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:57.029445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:57.029765 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 04:46:57.029929 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 04:46:57.030240 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":130,"threads_started":1}
I20260924 04:46:57.032603 3776 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34795
I20260924 04:46:57.032855 3776 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:57.033367 3776 heartbeater.cc:511] Master 127.0.98.62:34795 requested a full tablet report, sending...
I20260924 04:46:57.034401 3451 ts_manager.cc:194] Registered new tserver with Master: d2035222dc674a04917693e5dd4ddd85 (127.0.98.2:33773)
I20260924 04:46:57.034953 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:46:57.049994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34795
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:57.160405 3781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:57.160673 3781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:57.160717 3781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:57.164763 3781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:57.164829 3781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:57.164892 3781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:46:57.170254 3781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:57.171360 3781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:57.172395 3781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:57.182158 3789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:57.182161 3787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:57.182158 3786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:57.182372 3781 server_base.cc:1034] running on GCE node
I20260924 04:46:57.182746 3781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:57.183444 3781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:57.184602 3781 hybrid_clock.cc:648] HybridClock initialized: now 1790225217184585 us; error 41 us; skew 500 ppm
Sep 24 04:46:57 dist-test-slave-hvcl krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:57.187352 3781 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:46:57.188198 3781 webserver.cc:460] Webserver started at http://127.0.98.3:39653/ using document root <none> and password file <none>
I20260924 04:46:57.188720 3781 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:57.188781 3781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:57.188962 3781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:57.190902 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "3f8716456b82412486d4850fe39ce90a"
format_stamp: "Formatted at 2026-09-24 04:46:57 on dist-test-slave-hvcl"
server_key: "4e467b4ba9fb6c05f01cb40713133845"
server_key_iv: "143f1851723a57be5e51057a7bc5b1c3"
server_key_version: "encryptionkey@0"
I20260924 04:46:57.191462 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3f8716456b82412486d4850fe39ce90a"
format_stamp: "Formatted at 2026-09-24 04:46:57 on dist-test-slave-hvcl"
server_key: "4e467b4ba9fb6c05f01cb40713133845"
server_key_iv: "143f1851723a57be5e51057a7bc5b1c3"
server_key_version: "encryptionkey@0"
I20260924 04:46:57.195361 3781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:46:57.197711 3796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:57.198886 3781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:46:57.199010 3781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "3f8716456b82412486d4850fe39ce90a"
format_stamp: "Formatted at 2026-09-24 04:46:57 on dist-test-slave-hvcl"
server_key: "4e467b4ba9fb6c05f01cb40713133845"
server_key_iv: "143f1851723a57be5e51057a7bc5b1c3"
server_key_version: "encryptionkey@0"
I20260924 04:46:57.199100 3781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:57.212291 3781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:57.215459 3781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:57.215648 3781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:57.216234 3781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:57.217108 3781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:57.217156 3781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:57.217208 3781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:57.217236 3781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:57.225685 3781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35041
I20260924 04:46:57.225706 3909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35041 every 8 connection(s)
I20260924 04:46:57.226590 3781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:46:57 dist-test-slave-hvcl krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:57.236205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 3781
I20260924 04:46:57.236300 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:46:57.236562 392 external_mini_cluster.cc:1428] Setting key 646c516183d1462fda369e2d3939126f
I20260924 04:46:57.239976 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:57.228310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47581 (local address 127.0.98.62:34795)
0924 04:46:57.228471 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:57.228476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:57.229347 (+ 871us) server_negotiation.cc:407] Connection header received
0924 04:46:57.230687 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:57.230689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:57.230735 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:57.230809 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:57.232052 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:57.232618 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:57.233797 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:57.234015 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:57.234098 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:57.236561 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:57.236565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:57.236567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:57.236597 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:57.238084 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:57.238623 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:57.238627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:57.238629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:57.238708 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:57.239047 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:57.239050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:57.239051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:57.239229 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:57.239314 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:57.239745 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 04:46:57.239860 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:46:57.240830 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:57.228706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34795 (local address 127.0.98.3:47581)
0924 04:46:57.229191 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:57.229237 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:57.230481 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:57.230995 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:57.231005 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:57.231380 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:57.231898 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:57.231914 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:57.232746 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:57.232749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:57.233674 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:57.233684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:57.234174 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:57.234177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:57.234332 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:57.234917 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:57.234934 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:57.236396 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:57.238186 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:57.238192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:57.238208 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:57.238542 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:57.238805 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:57.238808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:57.238810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:57.238948 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:57.239313 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:57.239318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:57.239551 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 04:46:57.239699 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 04:46:57.239974 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":106,"threads_started":1}
I20260924 04:46:57.241917 3910 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34795
I20260924 04:46:57.242125 3910 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:57.242630 3910 heartbeater.cc:511] Master 127.0.98.62:34795 requested a full tablet report, sending...
I20260924 04:46:57.243613 3451 ts_manager.cc:194] Registered new tserver with Master: 3f8716456b82412486d4850fe39ce90a (127.0.98.3:35041)
I20260924 04:46:57.244149 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47581
I20260924 04:46:57.250267 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:46:57 dist-test-slave-hvcl krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225217, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 04:46:57 dist-test-slave-hvcl krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225217, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:57.274488 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:57.266365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.0.98.62:34795)
0924 04:46:57.266534 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:57.266539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:57.266679 (+ 140us) server_negotiation.cc:407] Connection header received
0924 04:46:57.266822 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:57.266825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:57.266869 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:57.266939 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:57.267660 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:57.268176 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:57.269007 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:57.269227 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:57.269314 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:57.271634 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:57.271637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:57.271639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:57.271663 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:57.272921 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:57.273341 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:57.273345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:57.273346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:57.273423 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:57.273652 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:57.273655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:57.273657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:57.273878 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:57.273995 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:57.274221 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 04:46:57.274336 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260924 04:46:57.274933 3451 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35594
I20260924 04:46:57.275955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 3513
I20260924 04:46:57.280999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 3647
I20260924 04:46:57.285652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 3781
I20260924 04:46:57.289898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 3419
2026-09-24T04:46:57Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2016 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:46:57 dist-test-slave-hvcl krb5kdc[3922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:46:57 dist-test-slave-hvcl krb5kdc[3922](info): set up 2 sockets
Sep 24 04:46:57 dist-test-slave-hvcl krb5kdc[3922](info): commencing operation
krb5kdc: starting...
W20260924 04:46:58.246783 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:46:58 dist-test-slave-hvcl krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:46:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:46:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:46:58.366641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36785
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:58.472853 3935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:58.473114 3935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:58.473168 3935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:58.477073 3935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:46:58.477129 3935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:58.477157 3935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:58.477177 3935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:46:58.477198 3935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:46:58.482379 3935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:58.483390 3935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:58.484354 3935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:58.492468 3941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:58.492436 3943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:58.492506 3940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:58.492674 3935 server_base.cc:1034] running on GCE node
I20260924 04:46:58.493311 3935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:58.494378 3935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:58.495527 3935 hybrid_clock.cc:648] HybridClock initialized: now 1790225218495508 us; error 32 us; skew 500 ppm
Sep 24 04:46:58 dist-test-slave-hvcl krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:58.498374 3935 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:46:58.499325 3935 webserver.cc:460] Webserver started at http://127.0.98.62:36655/ using document root <none> and password file <none>
I20260924 04:46:58.499845 3935 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:58.499907 3935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:58.500087 3935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:58.502053 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "ed45c0c10cf842f58b5e0bf90f36f716"
format_stamp: "Formatted at 2026-09-24 04:46:58 on dist-test-slave-hvcl"
server_key: "02e3d8967bda361ddcbb9bdbf94cd97e"
server_key_iv: "15ba20249799bb11cb7a56ace133a84b"
server_key_version: "encryptionkey@0"
I20260924 04:46:58.502583 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed45c0c10cf842f58b5e0bf90f36f716"
format_stamp: "Formatted at 2026-09-24 04:46:58 on dist-test-slave-hvcl"
server_key: "02e3d8967bda361ddcbb9bdbf94cd97e"
server_key_iv: "15ba20249799bb11cb7a56ace133a84b"
server_key_version: "encryptionkey@0"
I20260924 04:46:58.506069 3935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:46:58.508409 3950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:58.509645 3935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:46:58.509748 3935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "ed45c0c10cf842f58b5e0bf90f36f716"
format_stamp: "Formatted at 2026-09-24 04:46:58 on dist-test-slave-hvcl"
server_key: "02e3d8967bda361ddcbb9bdbf94cd97e"
server_key_iv: "15ba20249799bb11cb7a56ace133a84b"
server_key_version: "encryptionkey@0"
I20260924 04:46:58.509835 3935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:58.531049 3935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:58.534143 3935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:58.534339 3935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:58.541283 3935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36785
I20260924 04:46:58.541296 4002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36785 every 8 connection(s)
I20260924 04:46:58.542238 3935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:46:58.544994 4003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:46:58.550360 4003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716: Bootstrap starting.
I20260924 04:46:58.552573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 3935
I20260924 04:46:58.552698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:46:58.552891 4003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716: Neither blocks nor log segments found. Creating new log.
I20260924 04:46:58.552991 392 external_mini_cluster.cc:1428] Setting key 28c9f2bc51f01c37f691b1f1d366f354
I20260924 04:46:58.553684 4003 log.cc:824] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716: Log is configured to *not* fsync() on all Append() calls
I20260924 04:46:58.555642 4003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716: No bootstrap required, opened a new log
I20260924 04:46:58.559078 4003 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed45c0c10cf842f58b5e0bf90f36f716" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } }
I20260924 04:46:58.559294 4003 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:46:58.559346 4003 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed45c0c10cf842f58b5e0bf90f36f716, State: Initialized, Role: FOLLOWER
I20260924 04:46:58.559772 4003 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed45c0c10cf842f58b5e0bf90f36f716" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } }
I20260924 04:46:58.559901 4003 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:46:58.559965 4003 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:46:58.560056 4003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:46:58.561018 4003 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed45c0c10cf842f58b5e0bf90f36f716" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } }
Sep 24 04:46:58 dist-test-slave-hvcl krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:58.561362 4003 leader_election.cc:302] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed45c0c10cf842f58b5e0bf90f36f716; no voters:
I20260924 04:46:58.561707 4003 leader_election.cc:288] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:46:58.561789 4008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:46:58.562006 4008 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [term 1 LEADER]: Becoming Leader. State: Replica: ed45c0c10cf842f58b5e0bf90f36f716, State: Running, Role: LEADER
I20260924 04:46:58.562578 4008 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed45c0c10cf842f58b5e0bf90f36f716" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } }
I20260924 04:46:58.563035 4003 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:46:58.564136 4009 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed45c0c10cf842f58b5e0bf90f36f716" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed45c0c10cf842f58b5e0bf90f36f716" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } }
I20260924 04:46:58.564250 4009 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:58.564596 4010 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed45c0c10cf842f58b5e0bf90f36f716. Latest consensus state: current_term: 1 leader_uuid: "ed45c0c10cf842f58b5e0bf90f36f716" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed45c0c10cf842f58b5e0bf90f36f716" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } }
I20260924 04:46:58.564738 4010 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716 [sys.catalog]: This master's current role is: LEADER
I20260924 04:46:58.567350 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:58.554101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.0.98.62:36785)
0924 04:46:58.554464 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:58.554479 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:58.554514 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:46:58.555252 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:58.555271 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:58.555594 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:58.555933 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:58.556682 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:58.557674 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:58.558505 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:58.559010 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:58.559153 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:58.561773 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:58.561779 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:58.561795 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:58.561826 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:58.563791 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:58.564328 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:58.564334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:58.564340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:58.564471 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:58.564792 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:58.564795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:58.564796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:58.565313 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:58.566022 (+ 709us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:58.566146 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 04:46:58.566383 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":81,"threads_started":1}
W20260924 04:46:58.567807 4023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:46:58.567870 4023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:46:58.567929 4017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:46:58.568668 4017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:46:58.573787 4017 catalog_manager.cc:1347] Generated new cluster ID: 4317cfb6ca5d48f1b43f6a8644db40dd
I20260924 04:46:58.573850 4017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:46:58.611407 4017 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:46:58.612483 4017 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:46:58.619017 4017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed45c0c10cf842f58b5e0bf90f36f716: Generated new TSK 0
I20260924 04:46:58.619694 4017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:46:58.652379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36785
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:58.781646 4029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:58.781908 4029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:58.781951 4029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:58.785960 4029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:58.786026 4029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:58.786084 4029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:46:58.791299 4029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:58.792358 4029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:58.793371 4029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:58.802947 4034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:58.802930 4037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:58.803001 4035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:58.803254 4029 server_base.cc:1034] running on GCE node
I20260924 04:46:58.803822 4029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:58.804461 4029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:58.805640 4029 hybrid_clock.cc:648] HybridClock initialized: now 1790225218805619 us; error 33 us; skew 500 ppm
Sep 24 04:46:58 dist-test-slave-hvcl krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:58.808413 4029 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:46:58.809332 4029 webserver.cc:460] Webserver started at http://127.0.98.1:43265/ using document root <none> and password file <none>
I20260924 04:46:58.809875 4029 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:58.809937 4029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:58.810122 4029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:58.812199 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "e79fd1743e344f848b535d25fb75d451"
format_stamp: "Formatted at 2026-09-24 04:46:58 on dist-test-slave-hvcl"
server_key: "ad6f5f866939e5a92de8e6921ba49abb"
server_key_iv: "09c1fbfa19b7dfeceab0e651b2ff4ddd"
server_key_version: "encryptionkey@0"
I20260924 04:46:58.812747 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e79fd1743e344f848b535d25fb75d451"
format_stamp: "Formatted at 2026-09-24 04:46:58 on dist-test-slave-hvcl"
server_key: "ad6f5f866939e5a92de8e6921ba49abb"
server_key_iv: "09c1fbfa19b7dfeceab0e651b2ff4ddd"
server_key_version: "encryptionkey@0"
I20260924 04:46:58.816442 4029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:46:58.819006 4044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:58.820278 4029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:46:58.820394 4029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "e79fd1743e344f848b535d25fb75d451"
format_stamp: "Formatted at 2026-09-24 04:46:58 on dist-test-slave-hvcl"
server_key: "ad6f5f866939e5a92de8e6921ba49abb"
server_key_iv: "09c1fbfa19b7dfeceab0e651b2ff4ddd"
server_key_version: "encryptionkey@0"
I20260924 04:46:58.820492 4029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:58.832392 4029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:58.835434 4029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:58.835618 4029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:58.836184 4029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:58.836985 4029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:58.837031 4029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:58.837083 4029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:58.837114 4029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:58.845017 4029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38789
I20260924 04:46:58.845049 4157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38789 every 8 connection(s)
I20260924 04:46:58.845860 4029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:46:58.849295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 4029
I20260924 04:46:58.849428 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:46:58.849743 392 external_mini_cluster.cc:1428] Setting key 874575ac4313cf8307c2ccb8318eb091
Sep 24 04:46:58 dist-test-slave-hvcl krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:46:58.859992 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:58.847691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55021 (local address 127.0.98.62:36785)
0924 04:46:58.847835 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:58.847838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:58.848620 (+ 782us) server_negotiation.cc:407] Connection header received
0924 04:46:58.850262 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:58.850265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:58.850323 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:58.850394 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:58.851795 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:58.852320 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:58.853482 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:58.853827 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:58.853931 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:58.856511 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:58.856515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:58.856517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:58.856555 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:58.857968 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:58.858543 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:58.858546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:58.858548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:58.858631 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:58.858921 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:58.858924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:58.858925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:58.859150 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:58.859227 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:58.859745 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 04:46:58.859853 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:46:58.860860 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:58.848058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36785 (local address 127.0.98.1:55021)
0924 04:46:58.848480 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:58.848521 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:58.850041 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:58.850592 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:58.850600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:58.850945 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:58.851639 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:58.851659 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:58.852437 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:58.852441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:58.853369 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:58.853378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:58.853990 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:58.853993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:58.854141 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:58.854748 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:58.854764 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:58.856322 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:58.858088 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:58.858095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:58.858112 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:58.858450 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:58.858714 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:58.858716 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:58.858717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:58.858846 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:58.859236 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:58.859243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:58.859492 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 04:46:58.859690 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 04:46:58.859948 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":109,"threads_started":1}
I20260924 04:46:58.862075 4158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36785
I20260924 04:46:58.862350 4158 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:58.862828 4158 heartbeater.cc:511] Master 127.0.98.62:36785 requested a full tablet report, sending...
I20260924 04:46:58.864428 3967 ts_manager.cc:194] Registered new tserver with Master: e79fd1743e344f848b535d25fb75d451 (127.0.98.1:38789)
I20260924 04:46:58.865566 3967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:46:58.878235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36785
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:58.987702 4163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:58.987963 4163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:58.988005 4163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:58.992274 4163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:58.992350 4163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:58.992411 4163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:46:58.997748 4163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:58.998778 4163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:58.999935 4163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:46:59.009323 4168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:59.009433 4171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:59.009367 4163 server_base.cc:1034] running on GCE node
W20260924 04:46:59.009330 4169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:59.009991 4163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:59.010681 4163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:59.011842 4163 hybrid_clock.cc:648] HybridClock initialized: now 1790225219011822 us; error 30 us; skew 500 ppm
Sep 24 04:46:59 dist-test-slave-hvcl krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:59.014780 4163 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:46:59.015738 4163 webserver.cc:460] Webserver started at http://127.0.98.2:38915/ using document root <none> and password file <none>
I20260924 04:46:59.016294 4163 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:59.016358 4163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:59.016556 4163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:59.018644 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "a242abf17fdb4f1d95b80cbe01bf5c9f"
format_stamp: "Formatted at 2026-09-24 04:46:59 on dist-test-slave-hvcl"
server_key: "68d9a274a36557166643e394ff90b7f5"
server_key_iv: "4fcf4213611b74e11d2e1360a3cc0038"
server_key_version: "encryptionkey@0"
I20260924 04:46:59.019228 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a242abf17fdb4f1d95b80cbe01bf5c9f"
format_stamp: "Formatted at 2026-09-24 04:46:59 on dist-test-slave-hvcl"
server_key: "68d9a274a36557166643e394ff90b7f5"
server_key_iv: "4fcf4213611b74e11d2e1360a3cc0038"
server_key_version: "encryptionkey@0"
I20260924 04:46:59.022923 4163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:46:59.025492 4178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:59.026798 4163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:46:59.026906 4163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "a242abf17fdb4f1d95b80cbe01bf5c9f"
format_stamp: "Formatted at 2026-09-24 04:46:59 on dist-test-slave-hvcl"
server_key: "68d9a274a36557166643e394ff90b7f5"
server_key_iv: "4fcf4213611b74e11d2e1360a3cc0038"
server_key_version: "encryptionkey@0"
I20260924 04:46:59.027038 4163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:59.041700 4163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:59.044801 4163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:59.044948 4163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:59.045455 4163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:59.046223 4163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:59.046269 4163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:59.046319 4163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:59.046334 4163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:59.054754 4163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33713
I20260924 04:46:59.054788 4291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33713 every 8 connection(s)
I20260924 04:46:59.055722 4163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:46:59.064376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 4163
I20260924 04:46:59.064500 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:46:59.064761 392 external_mini_cluster.cc:1428] Setting key 42f3885e894f7d3c4c69c9bed5ba9ddf
Sep 24 04:46:59 dist-test-slave-hvcl krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:59.069145 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:59.057458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53547 (local address 127.0.98.62:36785)
0924 04:46:59.057598 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:59.057603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:59.058454 (+ 851us) server_negotiation.cc:407] Connection header received
0924 04:46:59.059837 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:59.059840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:59.059894 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:59.059968 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:59.061092 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:59.061601 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:59.062669 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:59.062937 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:59.063072 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:59.065619 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:59.065622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:59.065626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:59.065649 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:59.067147 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:59.067692 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:59.067695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:59.067696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:59.067761 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:59.068094 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:59.068098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:59.068099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:59.068279 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:59.068361 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:59.068853 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 04:46:59.068973 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:46:59.069936 4294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:59.057860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36785 (local address 127.0.98.2:53547)
0924 04:46:59.058306 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:59.058348 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:59.059647 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:59.060117 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:59.060124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:59.060465 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:59.060973 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:59.060986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:59.061698 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:59.061701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:59.062567 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:59.062576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:59.063098 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:59.063102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:59.063245 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:59.063815 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:59.063831 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:59.065458 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:59.067248 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:59.067254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:59.067271 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:59.067611 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:59.067846 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:59.067849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:59.067851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:59.067993 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:59.068374 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:59.068381 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:59.068633 (+ 252us) client_negotiation.cc:769] Sending connection context
0924 04:46:59.068798 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 04:46:59.069122 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":116,"threads_started":1}
I20260924 04:46:59.071071 4292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36785
I20260924 04:46:59.071358 4292 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:59.071987 4292 heartbeater.cc:511] Master 127.0.98.62:36785 requested a full tablet report, sending...
I20260924 04:46:59.073108 3967 ts_manager.cc:194] Registered new tserver with Master: a242abf17fdb4f1d95b80cbe01bf5c9f (127.0.98.2:33713)
I20260924 04:46:59.073709 3967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53547
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:46:59.093323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36785
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:46:59.202591 4297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:46:59.202843 4297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:46:59.202885 4297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:46:59.206867 4297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:46:59.206926 4297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:46:59.207007 4297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:46:59.212268 4297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:46:59.213300 4297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:46:59.214324 4297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:46:59.223824 4297 server_base.cc:1034] running on GCE node
W20260924 04:46:59.223810 4302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:59.223845 4303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:46:59.223807 4305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:46:59.224506 4297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:46:59.225199 4297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:46:59.226362 4297 hybrid_clock.cc:648] HybridClock initialized: now 1790225219226341 us; error 33 us; skew 500 ppm
Sep 24 04:46:59 dist-test-slave-hvcl krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:46:59.229075 4297 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:46:59.229928 4297 webserver.cc:460] Webserver started at http://127.0.98.3:34237/ using document root <none> and password file <none>
I20260924 04:46:59.230445 4297 fs_manager.cc:359] Metadata directory not provided
I20260924 04:46:59.230507 4297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:46:59.230751 4297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:46:59.232779 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "5306743134e241449374fc96ebc072a0"
format_stamp: "Formatted at 2026-09-24 04:46:59 on dist-test-slave-hvcl"
server_key: "27ef1ab397efcf9db6f9260808e416e1"
server_key_iv: "e633a45d0f6ec2b0b0c31ee0d95bfbf4"
server_key_version: "encryptionkey@0"
I20260924 04:46:59.233300 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5306743134e241449374fc96ebc072a0"
format_stamp: "Formatted at 2026-09-24 04:46:59 on dist-test-slave-hvcl"
server_key: "27ef1ab397efcf9db6f9260808e416e1"
server_key_iv: "e633a45d0f6ec2b0b0c31ee0d95bfbf4"
server_key_version: "encryptionkey@0"
I20260924 04:46:59.236820 4297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:46:59.239158 4312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:59.240372 4297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:46:59.240474 4297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "5306743134e241449374fc96ebc072a0"
format_stamp: "Formatted at 2026-09-24 04:46:59 on dist-test-slave-hvcl"
server_key: "27ef1ab397efcf9db6f9260808e416e1"
server_key_iv: "e633a45d0f6ec2b0b0c31ee0d95bfbf4"
server_key_version: "encryptionkey@0"
I20260924 04:46:59.240562 4297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:46:59.276970 4297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:46:59.280125 4297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:46:59.280300 4297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:46:59.280876 4297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:46:59.281687 4297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:46:59.281733 4297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:59.281785 4297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:46:59.281814 4297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:46:59.289948 4297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33441
I20260924 04:46:59.289965 4425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33441 every 8 connection(s)
I20260924 04:46:59.290838 4297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:46:59.299455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 4297
I20260924 04:46:59.299584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:46:59.299894 392 external_mini_cluster.cc:1428] Setting key 0dc53099bdc5e5b79cd30c2222ce3ccb
Sep 24 04:46:59 dist-test-slave-hvcl krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:59.304041 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:59.292546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44703 (local address 127.0.98.62:36785)
0924 04:46:59.292664 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:59.292668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:59.293572 (+ 904us) server_negotiation.cc:407] Connection header received
0924 04:46:59.294884 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:59.294888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:59.294941 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:59.295040 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:59.296227 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:59.296749 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:59.297806 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:59.298010 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:59.298074 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:59.300628 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:59.300631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:59.300634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:59.300660 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:59.302124 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:59.302596 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:59.302599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:59.302600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:59.302678 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:59.302942 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:59.302944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:59.302945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:59.303173 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:59.303249 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:59.303804 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 04:46:59.303910 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:46:59.304865 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:59.292975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36785 (local address 127.0.98.3:44703)
0924 04:46:59.293426 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 04:46:59.293465 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:46:59.294689 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:46:59.295234 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:46:59.295242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:46:59.295584 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:46:59.296094 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:59.296109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:59.296842 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:59.296844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:59.297717 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:46:59.297726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:46:59.298112 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:46:59.298115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:46:59.298250 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:59.298827 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:46:59.298843 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:46:59.300487 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:46:59.302234 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:59.302239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:59.302252 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:59.302520 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:59.302744 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:46:59.302747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:46:59.302748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:46:59.302877 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:46:59.303286 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:46:59.303292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:46:59.303541 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 04:46:59.303712 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 04:46:59.303990 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":115,"threads_started":1}
I20260924 04:46:59.305943 4426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36785
I20260924 04:46:59.306150 4426 heartbeater.cc:464] Registering TS with master...
I20260924 04:46:59.306627 4426 heartbeater.cc:511] Master 127.0.98.62:36785 requested a full tablet report, sending...
I20260924 04:46:59.307641 3967 ts_manager.cc:194] Registered new tserver with Master: 5306743134e241449374fc96ebc072a0 (127.0.98.3:33441)
I20260924 04:46:59.308182 3967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44703
I20260924 04:46:59.313571 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:46:59 dist-test-slave-hvcl krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:46:59 dist-test-slave-hvcl krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:46:59.337589 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:59.328527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.0.98.1:38789)
0924 04:46:59.328823 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:59.328837 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:59.328859 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:46:59.328910 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:59.328915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:59.329100 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:59.329222 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:59.329903 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:59.330708 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:59.331578 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:59.331974 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:59.332061 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:59.334317 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:59.334326 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:59.334340 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:59.334364 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:59.335920 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:59.336403 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:59.336408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:59.336409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:59.336485 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:59.336682 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:59.336684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:59.336685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:59.336908 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:59.337068 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:59.337267 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 04:46:59.337398 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":116,"threads_started":1}
Sep 24 04:46:59 dist-test-slave-hvcl krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:46:59.348405 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:46:59.340476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.0.98.62:36785)
0924 04:46:59.340614 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:46:59.340618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:46:59.340687 (+ 69us) server_negotiation.cc:407] Connection header received
0924 04:46:59.340822 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:46:59.340825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:46:59.340873 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:46:59.340935 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:46:59.341574 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:59.342103 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:59.342855 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:46:59.343145 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:46:59.343236 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:46:59.345460 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:46:59.345464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:46:59.345466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:46:59.345532 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:46:59.346787 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:59.347227 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:59.347231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:59.347233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:59.347312 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:46:59.347543 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:46:59.347546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:46:59.347548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:46:59.347798 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:46:59.347920 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 04:46:59.348134 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 04:46:59.348242 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 04:46:59.348840 3967 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60786
I20260924 04:46:59.349411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 4029
I20260924 04:46:59.354571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 4163
I20260924 04:46:59.359062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 4297
I20260924 04:46:59.363441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 3935
2026-09-24T04:46:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2073 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:46:59 dist-test-slave-hvcl krb5kdc[4438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:46:59 dist-test-slave-hvcl krb5kdc[4438](info): set up 2 sockets
Sep 24 04:46:59 dist-test-slave-hvcl krb5kdc[4438](info): commencing operation
krb5kdc: starting...
W20260924 04:47:00.319031 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:47:00 dist-test-slave-hvcl krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:47:00.441850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33039
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:00.550370 4451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:00.550661 4451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:00.550724 4451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:00.554824 4451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:00.554898 4451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:00.554924 4451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:00.554945 4451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:00.554986 4451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:47:00.560381 4451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:00.561448 4451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:00.562505 4451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:00.570825 4457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:00.570917 4456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:00.570907 4451 server_base.cc:1034] running on GCE node
W20260924 04:47:00.571107 4459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:00.571700 4451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:00.572857 4451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:00.574029 4451 hybrid_clock.cc:648] HybridClock initialized: now 1790225220574014 us; error 30 us; skew 500 ppm
Sep 24 04:47:00 dist-test-slave-hvcl krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:00.580856 4451 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:47:00.581615 4451 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:00.581748 4451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:00.581950 4451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:00.584103 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "8e20fe9ef1ff487681a590cad63b02e6"
format_stamp: "Formatted at 2026-09-24 04:47:00 on dist-test-slave-hvcl"
server_key: "358374ed397af963f5f0d08d666ee712"
server_key_iv: "e2fb53fd5c5496247eea00dee8bcdded"
server_key_version: "encryptionkey@0"
I20260924 04:47:00.584601 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "8e20fe9ef1ff487681a590cad63b02e6"
format_stamp: "Formatted at 2026-09-24 04:47:00 on dist-test-slave-hvcl"
server_key: "358374ed397af963f5f0d08d666ee712"
server_key_iv: "e2fb53fd5c5496247eea00dee8bcdded"
server_key_version: "encryptionkey@0"
I20260924 04:47:00.588491 4451 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:47:00.590830 4465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:00.592149 4451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:00.592252 4451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "8e20fe9ef1ff487681a590cad63b02e6"
format_stamp: "Formatted at 2026-09-24 04:47:00 on dist-test-slave-hvcl"
server_key: "358374ed397af963f5f0d08d666ee712"
server_key_iv: "e2fb53fd5c5496247eea00dee8bcdded"
server_key_version: "encryptionkey@0"
I20260924 04:47:00.592337 4451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:00.611935 4451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:00.615118 4451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:00.615272 4451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:00.621713 4451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33039
I20260924 04:47:00.621969 4517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33039 every 8 connection(s)
I20260924 04:47:00.622367 4451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:47:00.625092 4518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:00.627966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 4451
I20260924 04:47:00.628103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:47:00.628381 392 external_mini_cluster.cc:1428] Setting key 1fa95ec71350d349dfdafaa74c44cd38
I20260924 04:47:00.631250 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6: Bootstrap starting.
I20260924 04:47:00.633373 4518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:00.634085 4518 log.cc:824] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:00.635864 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6: No bootstrap required, opened a new log
Sep 24 04:47:00 dist-test-slave-hvcl krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:47:00.638943 4518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e20fe9ef1ff487681a590cad63b02e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33039 } }
I20260924 04:47:00.639202 4518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:00.639261 4518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e20fe9ef1ff487681a590cad63b02e6, State: Initialized, Role: FOLLOWER
I20260924 04:47:00.639699 4518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e20fe9ef1ff487681a590cad63b02e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33039 } }
I20260924 04:47:00.639823 4518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:47:00.639886 4518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:47:00.639977 4518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:00.640811 4518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e20fe9ef1ff487681a590cad63b02e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33039 } }
I20260924 04:47:00.641161 4518 leader_election.cc:302] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e20fe9ef1ff487681a590cad63b02e6; no voters:
I20260924 04:47:00.641518 4518 leader_election.cc:288] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:47:00.641779 4523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:00.642026 4523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [term 1 LEADER]: Becoming Leader. State: Replica: 8e20fe9ef1ff487681a590cad63b02e6, State: Running, Role: LEADER
I20260924 04:47:00.642400 4523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e20fe9ef1ff487681a590cad63b02e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33039 } }
I20260924 04:47:00.642670 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:00.629896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.0.98.62:33039)
0924 04:47:00.630346 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:00.630365 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:00.630401 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:47:00.631077 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:00.631096 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:00.631464 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:00.631788 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:00.632607 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:00.633626 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:00.634504 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:00.635003 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:00.635151 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:00.637400 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:00.637405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:00.637419 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:00.637445 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:00.639647 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:00.640099 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:00.640104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:00.640108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:00.640222 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:00.640476 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:00.640479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:00.640481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:00.640902 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:00.641059 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:00.641363 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 04:47:00.641649 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":125,"threads_started":1}
I20260924 04:47:00.642710 4518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:00.644073 4524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e20fe9ef1ff487681a590cad63b02e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e20fe9ef1ff487681a590cad63b02e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33039 } } }
I20260924 04:47:00.644186 4524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:00.644556 4525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e20fe9ef1ff487681a590cad63b02e6. Latest consensus state: current_term: 1 leader_uuid: "8e20fe9ef1ff487681a590cad63b02e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e20fe9ef1ff487681a590cad63b02e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33039 } } }
I20260924 04:47:00.644685 4525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:00.645449 4534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:00.648026 4534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:00.652988 4534 catalog_manager.cc:1347] Generated new cluster ID: 215573aab8d34d749833e763a0ece251
I20260924 04:47:00.653057 4534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:00.665520 4534 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:47:00.666512 4534 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:00.671999 4534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6: Generated new TSK 0
I20260924 04:47:00.672847 4534 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:47:00.702085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33039
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:00.815253 4544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:00.815557 4544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:00.815605 4544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:00.819875 4544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:00.819949 4544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:00.820014 4544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:47:00.825495 4544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:00.826579 4544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:00.827704 4544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:47:00.837769 4544 server_base.cc:1034] running on GCE node
W20260924 04:47:00.837755 4550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:00.837759 4549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:00.837989 4552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:00.838382 4544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:00.839063 4544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:00.840245 4544 hybrid_clock.cc:648] HybridClock initialized: now 1790225220840228 us; error 58 us; skew 500 ppm
Sep 24 04:47:00 dist-test-slave-hvcl krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:00.843096 4544 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:47:00.843953 4544 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:00.844024 4544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:00.844228 4544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:00.846319 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "735bd754f0a3447ab0ae1c13e236f1a3"
format_stamp: "Formatted at 2026-09-24 04:47:00 on dist-test-slave-hvcl"
server_key: "d36b184199022c6b58ac1e6ca4bb1ac4"
server_key_iv: "d2c212bd7cf08e3d24e47240b6febf3f"
server_key_version: "encryptionkey@0"
I20260924 04:47:00.846903 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "735bd754f0a3447ab0ae1c13e236f1a3"
format_stamp: "Formatted at 2026-09-24 04:47:00 on dist-test-slave-hvcl"
server_key: "d36b184199022c6b58ac1e6ca4bb1ac4"
server_key_iv: "d2c212bd7cf08e3d24e47240b6febf3f"
server_key_version: "encryptionkey@0"
I20260924 04:47:00.850718 4544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:47:00.853400 4558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:00.854734 4544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:47:00.854862 4544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "735bd754f0a3447ab0ae1c13e236f1a3"
format_stamp: "Formatted at 2026-09-24 04:47:00 on dist-test-slave-hvcl"
server_key: "d36b184199022c6b58ac1e6ca4bb1ac4"
server_key_iv: "d2c212bd7cf08e3d24e47240b6febf3f"
server_key_version: "encryptionkey@0"
I20260924 04:47:00.855023 4544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:00.873005 4544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:00.876346 4544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:00.876529 4544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:00.877117 4544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:00.877951 4544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:00.877998 4544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:00.878053 4544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:00.878080 4544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:00.886448 4544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44999
I20260924 04:47:00.886515 4671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44999 every 8 connection(s)
I20260924 04:47:00.887089 4544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:47:00.888375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 4544
I20260924 04:47:00.888484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:47:00.888876 392 external_mini_cluster.cc:1428] Setting key f941326bb3280641728634468e9130ee
Sep 24 04:47:00 dist-test-slave-hvcl krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:47:00.901454 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:00.888818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47409 (local address 127.0.98.62:33039)
0924 04:47:00.888970 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:00.888974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:00.889868 (+ 894us) server_negotiation.cc:407] Connection header received
0924 04:47:00.891512 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:00.891514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:00.891557 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:00.891622 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:00.892891 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:00.893459 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:00.895121 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:00.895354 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:00.895453 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:00.898121 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:00.898124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:00.898126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:00.898149 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:00.899433 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:00.899916 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:00.899919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:00.899920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:00.900008 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:00.900317 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:00.900319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:00.900320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:00.900494 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:00.900567 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:00.901143 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 04:47:00.901271 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:47:00.902256 4674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:00.889287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33039 (local address 127.0.98.1:47409)
0924 04:47:00.889738 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:00.889781 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:00.891287 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:00.891824 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:00.891833 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:00.892268 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:00.892749 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:00.892763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:00.893580 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:00.893583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:00.895009 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:00.895019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:00.895499 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:00.895503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:00.895711 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:00.896341 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:00.896361 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:00.897962 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:00.899547 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:00.899552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:00.899565 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:00.899810 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:00.900088 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:00.900090 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:00.900091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:00.900236 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:00.900570 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:00.900575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:00.900837 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 04:47:00.901099 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 04:47:00.901389 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260924 04:47:00.903472 4672 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33039
I20260924 04:47:00.903713 4672 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:00.904150 4672 heartbeater.cc:511] Master 127.0.98.62:33039 requested a full tablet report, sending...
I20260924 04:47:00.905654 4482 ts_manager.cc:194] Registered new tserver with Master: 735bd754f0a3447ab0ae1c13e236f1a3 (127.0.98.1:44999)
I20260924 04:47:00.906818 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:47:00.919412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33039
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:01.029039 4677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:01.029336 4677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:01.029402 4677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:01.033552 4677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:01.033620 4677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:01.033707 4677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:47:01.039044 4677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:01.040189 4677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:01.041200 4677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:01.050590 4683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:01.050611 4685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:01.050591 4682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:01.050882 4677 server_base.cc:1034] running on GCE node
I20260924 04:47:01.051441 4677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:01.052093 4677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:01.053272 4677 hybrid_clock.cc:648] HybridClock initialized: now 1790225221053251 us; error 31 us; skew 500 ppm
Sep 24 04:47:01 dist-test-slave-hvcl krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:01.056236 4677 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:47:01.056981 4677 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:01.057046 4677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:01.057235 4677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:01.059412 4677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "4db0dc8f77c74265ae0ccec6b26e10b6"
format_stamp: "Formatted at 2026-09-24 04:47:01 on dist-test-slave-hvcl"
server_key: "22b8f120ef7dc0734603190b029dfac8"
server_key_iv: "32df4e110e07eaffacebb261fc2545cb"
server_key_version: "encryptionkey@0"
I20260924 04:47:01.060047 4677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4db0dc8f77c74265ae0ccec6b26e10b6"
format_stamp: "Formatted at 2026-09-24 04:47:01 on dist-test-slave-hvcl"
server_key: "22b8f120ef7dc0734603190b029dfac8"
server_key_iv: "32df4e110e07eaffacebb261fc2545cb"
server_key_version: "encryptionkey@0"
I20260924 04:47:01.063791 4677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:47:01.066285 4691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:01.067663 4677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:01.067762 4677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "4db0dc8f77c74265ae0ccec6b26e10b6"
format_stamp: "Formatted at 2026-09-24 04:47:01 on dist-test-slave-hvcl"
server_key: "22b8f120ef7dc0734603190b029dfac8"
server_key_iv: "32df4e110e07eaffacebb261fc2545cb"
server_key_version: "encryptionkey@0"
I20260924 04:47:01.067869 4677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:01.085327 4677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:01.088461 4677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:01.088639 4677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:01.089226 4677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:01.090066 4677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:01.090109 4677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:01.090149 4677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:01.090178 4677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:01.098402 4677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36089
I20260924 04:47:01.098448 4804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36089 every 8 connection(s)
I20260924 04:47:01.098959 4677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:47:01.105664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 4677
I20260924 04:47:01.105799 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:47:01.106086 392 external_mini_cluster.cc:1428] Setting key 0892db0ac557ea596c29332128b7d0e2
Sep 24 04:47:01 dist-test-slave-hvcl krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:47:01.112882 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.100687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35555 (local address 127.0.98.62:33039)
0924 04:47:01.100852 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:01.100856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:01.101724 (+ 868us) server_negotiation.cc:407] Connection header received
0924 04:47:01.103092 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:01.103096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:01.103151 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:01.103225 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:01.104388 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.104926 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.106112 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.106362 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.106462 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.109118 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:01.109121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:01.109123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:01.109146 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:01.110601 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:01.111167 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:01.111171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:01.111172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:01.111265 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:01.111647 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:01.111651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:01.111652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:01.111942 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:01.112048 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:01.112568 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 04:47:01.112700 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 04:47:01.113840 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.101138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33039 (local address 127.0.98.2:35555)
0924 04:47:01.101575 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:01.101616 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:01.102844 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:01.103403 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:01.103412 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:01.103764 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:01.104246 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:01.104260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.105042 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.105046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:01.106003 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:01.106012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.106502 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.106506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:01.106688 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.107301 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:01.107317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:01.108955 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:01.110721 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:01.110726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:01.110738 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:01.111079 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:01.111378 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:01.111381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:01.111382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:01.111529 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:01.112056 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:01.112061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:01.112319 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 04:47:01.112475 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 04:47:01.112804 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:47:01.115154 4805 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33039
I20260924 04:47:01.115396 4805 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:01.115845 4805 heartbeater.cc:511] Master 127.0.98.62:33039 requested a full tablet report, sending...
I20260924 04:47:01.116873 4482 ts_manager.cc:194] Registered new tserver with Master: 4db0dc8f77c74265ae0ccec6b26e10b6 (127.0.98.2:36089)
I20260924 04:47:01.117477 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:47:01.135022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33039
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:01.245151 4810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:01.245455 4810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:01.245502 4810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:01.250018 4810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:01.250090 4810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:01.250154 4810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:47:01.256088 4810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:01.257189 4810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:01.258453 4810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:01.269112 4816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:01.269112 4815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:01.269184 4810 server_base.cc:1034] running on GCE node
W20260924 04:47:01.269112 4818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:01.269763 4810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:01.270434 4810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:01.271587 4810 hybrid_clock.cc:648] HybridClock initialized: now 1790225221271567 us; error 38 us; skew 500 ppm
Sep 24 04:47:01 dist-test-slave-hvcl krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:01.274421 4810 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:47:01.275277 4810 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:01.275352 4810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:01.275561 4810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:01.277669 4810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "0767ff1671514446aed5c1f2f53453dd"
format_stamp: "Formatted at 2026-09-24 04:47:01 on dist-test-slave-hvcl"
server_key: "8e5ecb23d0875c308baf07c30b7721ee"
server_key_iv: "6590e1dee92ea5a1ee0fa02303e53b43"
server_key_version: "encryptionkey@0"
I20260924 04:47:01.278231 4810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0767ff1671514446aed5c1f2f53453dd"
format_stamp: "Formatted at 2026-09-24 04:47:01 on dist-test-slave-hvcl"
server_key: "8e5ecb23d0875c308baf07c30b7721ee"
server_key_iv: "6590e1dee92ea5a1ee0fa02303e53b43"
server_key_version: "encryptionkey@0"
I20260924 04:47:01.282140 4810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:47:01.284819 4824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:01.286194 4810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:47:01.286309 4810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "0767ff1671514446aed5c1f2f53453dd"
format_stamp: "Formatted at 2026-09-24 04:47:01 on dist-test-slave-hvcl"
server_key: "8e5ecb23d0875c308baf07c30b7721ee"
server_key_iv: "6590e1dee92ea5a1ee0fa02303e53b43"
server_key_version: "encryptionkey@0"
I20260924 04:47:01.286417 4810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:01.298137 4810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:01.301144 4810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:01.301329 4810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:01.301885 4810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:01.302702 4810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:01.302749 4810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:01.302801 4810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:01.302829 4810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:01.311437 4810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39409
I20260924 04:47:01.311463 4937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39409 every 8 connection(s)
I20260924 04:47:01.312021 4810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:47:01.321195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 4810
I20260924 04:47:01.321326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:47:01.321640 392 external_mini_cluster.cc:1428] Setting key a474e109faad761aa1852de9215d0bc4
Sep 24 04:47:01 dist-test-slave-hvcl krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:47:01.326648 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.313959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56741 (local address 127.0.98.62:33039)
0924 04:47:01.314120 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:01.314124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:01.315001 (+ 877us) server_negotiation.cc:407] Connection header received
0924 04:47:01.316378 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:01.316382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:01.316439 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:01.316510 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:01.317738 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.318265 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.319637 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.319862 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.319956 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.322915 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:01.322919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:01.322922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:01.322952 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:01.324427 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:01.325027 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:01.325030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:01.325032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:01.325109 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:01.325511 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:01.325514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:01.325515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:01.325701 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:01.325775 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:01.326422 (+ 647us) server_negotiation.cc:299] Negotiation successful
0924 04:47:01.326527 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 04:47:01.327586 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.314288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33039 (local address 127.0.98.3:56741)
0924 04:47:01.314766 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:01.314864 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:01.316162 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:01.316698 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:01.316705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:01.317085 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:01.317586 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:01.317599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.318528 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.318532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:01.319507 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:01.319518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.320014 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.320018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:01.320180 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.320801 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:01.320818 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:01.322742 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:01.324580 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:01.324585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:01.324600 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:01.324929 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:01.325233 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:01.325236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:01.325238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:01.325422 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:01.325835 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:01.325840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:01.326195 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 04:47:01.326369 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 04:47:01.326655 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":110,"threads_started":1}
I20260924 04:47:01.328665 4938 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33039
I20260924 04:47:01.328959 4938 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:01.329434 4938 heartbeater.cc:511] Master 127.0.98.62:33039 requested a full tablet report, sending...
I20260924 04:47:01.330538 4482 ts_manager.cc:194] Registered new tserver with Master: 0767ff1671514446aed5c1f2f53453dd (127.0.98.3:39409)
I20260924 04:47:01.331209 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56741
I20260924 04:47:01.335906 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:47:01.345712 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.337733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.0.98.62:33039)
0924 04:47:01.337894 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:01.337898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:01.337997 (+ 99us) server_negotiation.cc:407] Connection header received
0924 04:47:01.338147 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:01.338150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:01.338194 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:01.338267 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:01.339053 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.339606 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.340477 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.340694 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.340807 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.342217 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:01.342220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:01.342222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:01.342247 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:01.343818 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:01.344312 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:01.344315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:01.344316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:01.344412 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:01.344839 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:01.344842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:01.344843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:01.345098 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:01.345215 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:01.345479 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 04:47:01.345587 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 04:47:01.348873 4482 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:47:01.352214 4482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:47:01.367063 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.361263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36089 (local address 127.0.0.1:39418)
0924 04:47:01.362182 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:01.362195 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:01.362281 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:01.363189 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:01.363196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:01.363219 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:01.363371 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:01.363379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.364488 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.364491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:01.365622 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:01.365631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.366472 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.366475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:01.366584 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.366622 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 04:47:01.366741 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 04:47:01.366874 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":137,"threads_started":1}
I20260924 04:47:01.367085 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.361993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39418 (local address 127.0.98.2:36089)
0924 04:47:01.362673 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:01.362685 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:01.362710 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:47:01.362785 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:01.362790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:01.363017 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:01.363507 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:01.363645 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.364371 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.365735 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.366360 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.366465 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.366608 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:01.366773 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 04:47:01.366873 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":100,"threads_started":1}
I20260924 04:47:01.367887 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.362002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44999 (local address 127.0.0.1:46402)
0924 04:47:01.362346 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:01.362359 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:01.362446 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:01.363340 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:01.363343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:01.363361 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:01.363436 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:01.363441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.364909 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.364912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:01.366112 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:01.366121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.367139 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.367142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:01.367550 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.367591 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 04:47:01.367650 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 04:47:01.367748 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":91,"thread_start_us":96,"threads_started":1}
I20260924 04:47:01.367992 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.361267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49184 (local address 127.0.98.3:39409)
0924 04:47:01.362824 (+ 1557us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:01.362834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:01.362852 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:47:01.362914 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:01.362918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:01.363110 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:01.363236 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:01.363581 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.364757 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.366273 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.367062 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.367170 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.367307 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:01.367753 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 04:47:01.367849 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1468,"thread_start_us":139,"threads_started":1}
I20260924 04:47:01.368204 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.361114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39409 (local address 127.0.0.1:49184)
0924 04:47:01.361667 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:01.361717 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:01.361913 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:01.363397 (+ 1484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:01.363400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:01.363416 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:01.363490 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:01.363495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.365101 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.365104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:01.366186 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:01.366193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.367371 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.367375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:01.367547 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.367586 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 04:47:01.367653 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:47:01.368123 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":423,"thread_start_us":89,"threads_started":1}
I20260924 04:47:01.368544 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.362412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.0.98.1:44999)
0924 04:47:01.362709 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:01.362714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:01.362734 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:47:01.362794 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:01.362798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:01.362996 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:01.363165 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:01.363545 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.364773 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.366233 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.367007 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.367453 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.367811 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:01.367861 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 04:47:01.368404 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
I20260924 04:47:01.370538 4739 tablet_service.cc:1467] Processing CreateTablet for tablet dbfa57a5b85445819af904efdf9548a6 (DEFAULT_TABLE table=test-table [id=ece933d63961426298f953175ae50dfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:01.370947 4872 tablet_service.cc:1467] Processing CreateTablet for tablet dbfa57a5b85445819af904efdf9548a6 (DEFAULT_TABLE table=test-table [id=ece933d63961426298f953175ae50dfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:01.370947 4606 tablet_service.cc:1467] Processing CreateTablet for tablet dbfa57a5b85445819af904efdf9548a6 (DEFAULT_TABLE table=test-table [id=ece933d63961426298f953175ae50dfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:01.371615 4739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfa57a5b85445819af904efdf9548a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:01.371989 4606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfa57a5b85445819af904efdf9548a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:01.371995 4872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfa57a5b85445819af904efdf9548a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:01.378705 4955 tablet_bootstrap.cc:492] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6: Bootstrap starting.
I20260924 04:47:01.379359 4957 tablet_bootstrap.cc:492] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd: Bootstrap starting.
I20260924 04:47:01.379779 4956 tablet_bootstrap.cc:492] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3: Bootstrap starting.
I20260924 04:47:01.381201 4955 tablet_bootstrap.cc:654] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:01.381479 4956 tablet_bootstrap.cc:654] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:01.381475 4957 tablet_bootstrap.cc:654] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:01.382441 4956 log.cc:824] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:01.382498 4957 log.cc:824] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:01.382488 4955 log.cc:824] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:01.384564 4957 tablet_bootstrap.cc:492] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd: No bootstrap required, opened a new log
I20260924 04:47:01.384564 4955 tablet_bootstrap.cc:492] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6: No bootstrap required, opened a new log
I20260924 04:47:01.384564 4956 tablet_bootstrap.cc:492] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3: No bootstrap required, opened a new log
I20260924 04:47:01.384756 4957 ts_tablet_manager.cc:1397] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:47:01.384756 4955 ts_tablet_manager.cc:1397] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 04:47:01.384756 4956 ts_tablet_manager.cc:1397] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:47:01.387737 4957 raft_consensus.cc:348] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0767ff1671514446aed5c1f2f53453dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39409 } } peers { permanent_uuid: "4db0dc8f77c74265ae0ccec6b26e10b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "735bd754f0a3447ab0ae1c13e236f1a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44999 } }
I20260924 04:47:01.387984 4957 raft_consensus.cc:374] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:01.388053 4957 raft_consensus.cc:729] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0767ff1671514446aed5c1f2f53453dd, State: Initialized, Role: FOLLOWER
I20260924 04:47:01.388509 4957 consensus_queue.cc:260] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0767ff1671514446aed5c1f2f53453dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39409 } } peers { permanent_uuid: "4db0dc8f77c74265ae0ccec6b26e10b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "735bd754f0a3447ab0ae1c13e236f1a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44999 } }
I20260924 04:47:01.388639 4956 raft_consensus.cc:348] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0767ff1671514446aed5c1f2f53453dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39409 } } peers { permanent_uuid: "4db0dc8f77c74265ae0ccec6b26e10b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "735bd754f0a3447ab0ae1c13e236f1a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44999 } }
I20260924 04:47:01.388639 4955 raft_consensus.cc:348] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0767ff1671514446aed5c1f2f53453dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39409 } } peers { permanent_uuid: "4db0dc8f77c74265ae0ccec6b26e10b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "735bd754f0a3447ab0ae1c13e236f1a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44999 } }
I20260924 04:47:01.388906 4955 raft_consensus.cc:374] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:01.388995 4956 raft_consensus.cc:374] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:01.389089 4956 raft_consensus.cc:729] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 735bd754f0a3447ab0ae1c13e236f1a3, State: Initialized, Role: FOLLOWER
I20260924 04:47:01.389176 4938 heartbeater.cc:503] Master 127.0.98.62:33039 was elected leader, sending a full tablet report...
I20260924 04:47:01.389473 4957 ts_tablet_manager.cc:1428] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:47:01.388989 4955 raft_consensus.cc:729] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db0dc8f77c74265ae0ccec6b26e10b6, State: Initialized, Role: FOLLOWER
I20260924 04:47:01.389667 4956 consensus_queue.cc:260] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0767ff1671514446aed5c1f2f53453dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39409 } } peers { permanent_uuid: "4db0dc8f77c74265ae0ccec6b26e10b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "735bd754f0a3447ab0ae1c13e236f1a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44999 } }
I20260924 04:47:01.390106 4955 consensus_queue.cc:260] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0767ff1671514446aed5c1f2f53453dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39409 } } peers { permanent_uuid: "4db0dc8f77c74265ae0ccec6b26e10b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "735bd754f0a3447ab0ae1c13e236f1a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44999 } }
I20260924 04:47:01.390378 4672 heartbeater.cc:503] Master 127.0.98.62:33039 was elected leader, sending a full tablet report...
I20260924 04:47:01.391192 4955 ts_tablet_manager.cc:1428] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:47:01.391464 4956 ts_tablet_manager.cc:1428] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:47:01.391651 4805 heartbeater.cc:503] Master 127.0.98.62:33039 was elected leader, sending a full tablet report...
I20260924 04:47:01.554539 4961 raft_consensus.cc:482] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:47:01.554765 4961 raft_consensus.cc:504] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0767ff1671514446aed5c1f2f53453dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39409 } } peers { permanent_uuid: "4db0dc8f77c74265ae0ccec6b26e10b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "735bd754f0a3447ab0ae1c13e236f1a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44999 } }
I20260924 04:47:01.555989 4961 leader_election.cc:288] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4db0dc8f77c74265ae0ccec6b26e10b6 (127.0.98.2:36089), 735bd754f0a3447ab0ae1c13e236f1a3 (127.0.98.1:44999)
I20260924 04:47:01.560859 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.556344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51793 (local address 127.0.98.1:44999)
0924 04:47:01.556511 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:01.556515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:01.556791 (+ 276us) server_negotiation.cc:407] Connection header received
0924 04:47:01.556882 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:01.556885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:01.556931 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:01.556996 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:01.557219 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.557932 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.559541 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.560156 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.560266 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.560329 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:01.560638 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 04:47:01.560721 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 04:47:01.560883 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.556259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36089 (local address 127.0.98.3:52647)
0924 04:47:01.556459 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:01.556475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:01.556570 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:01.556884 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:01.556886 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:01.556899 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:01.557003 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:01.557009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.557997 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.558001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:01.559422 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:01.559433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.560301 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.560304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:01.560468 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.560551 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 04:47:01.560620 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:47:01.560724 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":113}
I20260924 04:47:01.560912 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.556421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52647 (local address 127.0.98.2:36089)
0924 04:47:01.556521 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:01.556525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:01.556540 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:47:01.556714 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:01.556717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:01.556772 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:01.556862 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:01.557109 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.557863 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.559541 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.560166 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.560266 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.560329 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:01.560728 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 04:47:01.560802 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:47:01.560890 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.556295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44999 (local address 127.0.98.3:51793)
0924 04:47:01.556691 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:01.556702 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:01.556786 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:01.557017 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:01.557020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:01.557033 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:01.557126 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:01.557131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.558031 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.558034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:01.559422 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:01.559433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.560284 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.560289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:01.560489 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.560543 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 04:47:01.560627 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:47:01.560728 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":139,"thread_start_us":112,"threads_started":1}
I20260924 04:47:01.561537 4759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfa57a5b85445819af904efdf9548a6" candidate_uuid: "0767ff1671514446aed5c1f2f53453dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db0dc8f77c74265ae0ccec6b26e10b6" is_pre_election: true
I20260924 04:47:01.561888 4759 raft_consensus.cc:2454] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0767ff1671514446aed5c1f2f53453dd in term 0.
I20260924 04:47:01.561843 4626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfa57a5b85445819af904efdf9548a6" candidate_uuid: "0767ff1671514446aed5c1f2f53453dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "735bd754f0a3447ab0ae1c13e236f1a3" is_pre_election: true
I20260924 04:47:01.562131 4626 raft_consensus.cc:2454] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0767ff1671514446aed5c1f2f53453dd in term 0.
I20260924 04:47:01.562415 4828 leader_election.cc:302] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0767ff1671514446aed5c1f2f53453dd, 4db0dc8f77c74265ae0ccec6b26e10b6; no voters:
I20260924 04:47:01.562703 4961 raft_consensus.cc:2781] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:47:01.562798 4961 raft_consensus.cc:482] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 04:47:01.562815 4939 tablet.cc:2369] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:47:01.562848 4961 raft_consensus.cc:3037] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:01.564016 4961 raft_consensus.cc:504] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0767ff1671514446aed5c1f2f53453dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39409 } } peers { permanent_uuid: "4db0dc8f77c74265ae0ccec6b26e10b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "735bd754f0a3447ab0ae1c13e236f1a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44999 } }
I20260924 04:47:01.564324 4961 leader_election.cc:288] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [CANDIDATE]: Term 1 election: Requested vote from peers 4db0dc8f77c74265ae0ccec6b26e10b6 (127.0.98.2:36089), 735bd754f0a3447ab0ae1c13e236f1a3 (127.0.98.1:44999)
I20260924 04:47:01.564697 4759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfa57a5b85445819af904efdf9548a6" candidate_uuid: "0767ff1671514446aed5c1f2f53453dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db0dc8f77c74265ae0ccec6b26e10b6"
I20260924 04:47:01.564734 4626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfa57a5b85445819af904efdf9548a6" candidate_uuid: "0767ff1671514446aed5c1f2f53453dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "735bd754f0a3447ab0ae1c13e236f1a3"
I20260924 04:47:01.564815 4759 raft_consensus.cc:3037] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:01.564845 4626 raft_consensus.cc:3037] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:01.565797 4626 raft_consensus.cc:2454] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0767ff1671514446aed5c1f2f53453dd in term 1.
I20260924 04:47:01.565840 4759 raft_consensus.cc:2454] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0767ff1671514446aed5c1f2f53453dd in term 1.
I20260924 04:47:01.566134 4828 leader_election.cc:302] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0767ff1671514446aed5c1f2f53453dd, 4db0dc8f77c74265ae0ccec6b26e10b6, 735bd754f0a3447ab0ae1c13e236f1a3; no voters:
I20260924 04:47:01.566286 4961 raft_consensus.cc:2781] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:01.566501 4961 raft_consensus.cc:686] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [term 1 LEADER]: Becoming Leader. State: Replica: 0767ff1671514446aed5c1f2f53453dd, State: Running, Role: LEADER
I20260924 04:47:01.566848 4961 consensus_queue.cc:237] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0767ff1671514446aed5c1f2f53453dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39409 } } peers { permanent_uuid: "4db0dc8f77c74265ae0ccec6b26e10b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "735bd754f0a3447ab0ae1c13e236f1a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44999 } }
I20260924 04:47:01.569895 4478 catalog_manager.cc:5495] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0767ff1671514446aed5c1f2f53453dd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0767ff1671514446aed5c1f2f53453dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0767ff1671514446aed5c1f2f53453dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4db0dc8f77c74265ae0ccec6b26e10b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "735bd754f0a3447ab0ae1c13e236f1a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44999 } health_report { overall_health: UNKNOWN } } }
W20260924 04:47:01.600008 4806 tablet.cc:2369] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:47:01.605626 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.601766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.0.98.3:39409)
0924 04:47:01.601902 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:01.601907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:01.601942 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:47:01.602115 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:01.602118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:01.602176 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:01.602302 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:47:01.602520 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.603061 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.604098 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:01.604343 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:01.604436 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:01.604736 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:47:01.605196 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:47:01.605318 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:01.605388 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 04:47:01.605450 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:47:01.611574 4626 raft_consensus.cc:1261] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3 [term 1 FOLLOWER]: Refusing update from remote peer 0767ff1671514446aed5c1f2f53453dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:01.611574 4759 raft_consensus.cc:1261] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6 [term 1 FOLLOWER]: Refusing update from remote peer 0767ff1671514446aed5c1f2f53453dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:01.612236 4965 consensus_queue.cc:1035] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "735bd754f0a3447ab0ae1c13e236f1a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:01.612406 4961 consensus_queue.cc:1035] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4db0dc8f77c74265ae0ccec6b26e10b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:01.631778 4482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:01.631968 4482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:01.634528 4482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e20fe9ef1ff487681a590cad63b02e6: Sending DeleteTablet for 3 replicas of tablet dbfa57a5b85445819af904efdf9548a6
I20260924 04:47:01.635368 4739 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfa57a5b85445819af904efdf9548a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39418
I20260924 04:47:01.635471 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfa57a5b85445819af904efdf9548a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49184
I20260924 04:47:01.635604 4606 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfa57a5b85445819af904efdf9548a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46402
I20260924 04:47:01.636098 4980 tablet_replica.cc:330] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3: stopping tablet replica
I20260924 04:47:01.636186 4978 tablet_replica.cc:330] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6: stopping tablet replica
I20260924 04:47:01.636456 4980 raft_consensus.cc:2229] T dbfa57a5b85445819af904efdf9548a6 P 735bd754f0a3447ab0ae1c13e236f1a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:01.636456 4978 raft_consensus.cc:2229] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:01.636484 4979 tablet_replica.cc:330] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd: stopping tablet replica
I20260924 04:47:01.636775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 4544
I20260924 04:47:01.637461 4979 raft_consensus.cc:2229] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:47:01.637733 4978 raft_consensus.cc:2258] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:01.637928 4979 raft_consensus.cc:2258] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:01.639428 4978 ts_tablet_manager.cc:1905] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:01.639428 4979 ts_tablet_manager.cc:1905] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:01.642030 4978 ts_tablet_manager.cc:1918] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:01.642030 4979 ts_tablet_manager.cc:1918] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:01.642130 4978 log.cc:1196] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/dbfa57a5b85445819af904efdf9548a6
I20260924 04:47:01.642249 4979 log.cc:1196] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/dbfa57a5b85445819af904efdf9548a6
I20260924 04:47:01.642412 4978 ts_tablet_manager.cc:1939] T dbfa57a5b85445819af904efdf9548a6 P 4db0dc8f77c74265ae0ccec6b26e10b6: Deleting consensus metadata
I20260924 04:47:01.642671 4979 ts_tablet_manager.cc:1939] T dbfa57a5b85445819af904efdf9548a6 P 0767ff1671514446aed5c1f2f53453dd: Deleting consensus metadata
W20260924 04:47:01.643396 4467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44999 (error 108)
I20260924 04:47:01.643858 4469 catalog_manager.cc:4841] TS 4db0dc8f77c74265ae0ccec6b26e10b6 (127.0.98.2:36089): tablet dbfa57a5b85445819af904efdf9548a6 (table test-table [id=ece933d63961426298f953175ae50dfb]) successfully deleted
I20260924 04:47:01.644058 4468 catalog_manager.cc:4841] TS 0767ff1671514446aed5c1f2f53453dd (127.0.98.3:39409): tablet dbfa57a5b85445819af904efdf9548a6 (table test-table [id=ece933d63961426298f953175ae50dfb]) successfully deleted
I20260924 04:47:01.644335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 4677
I20260924 04:47:01.645006 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.644657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44999 (local address 127.0.0.1:46410)
0924 04:47:01.644848 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:01.644921 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260924 04:47:01.645258 4467 catalog_manager.cc:4584] TS 735bd754f0a3447ab0ae1c13e236f1a3 (127.0.98.1:44999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbfa57a5b85445819af904efdf9548a6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44999: connect: Connection refused (error 111)
I20260924 04:47:01.649605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 4810
I20260924 04:47:01.654282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 4451
2026-09-24T04:47:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2291 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T04:47:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:01Z Disabled control of system clock
I20260924 04:47:01.682014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40351
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:47:01.791106 4985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:01.791388 4985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:01.791424 4985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:01.795449 4985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:01.795513 4985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:01.795527 4985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:01.795544 4985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:01.795560 4985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:47:01.800382 4985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40351
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:01.801452 4985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:01.802788 4985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:01.811367 4990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:01.811367 4993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:01.811494 4991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:01.811545 4985 server_base.cc:1034] running on GCE node
I20260924 04:47:01.812322 4985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:01.813444 4985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:01.814620 4985 hybrid_clock.cc:648] HybridClock initialized: now 1790225221814597 us; error 33 us; skew 500 ppm
I20260924 04:47:01.816567 4985 webserver.cc:460] Webserver started at http://127.0.98.62:39275/ using document root <none> and password file <none>
I20260924 04:47:01.817122 4985 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:01.817186 4985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:01.817368 4985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:01.819425 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "fd6fd1bb4c484622946f5d36d859de84"
format_stamp: "Formatted at 2026-09-24 04:47:01 on dist-test-slave-hvcl"
server_key: "f9616b2f79f6df03a1de27e3d0753ac5"
server_key_iv: "7972089960ca67f5b03fe4ea96d84091"
server_key_version: "encryptionkey@0"
I20260924 04:47:01.819955 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "fd6fd1bb4c484622946f5d36d859de84"
format_stamp: "Formatted at 2026-09-24 04:47:01 on dist-test-slave-hvcl"
server_key: "f9616b2f79f6df03a1de27e3d0753ac5"
server_key_iv: "7972089960ca67f5b03fe4ea96d84091"
server_key_version: "encryptionkey@0"
I20260924 04:47:01.823603 4985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:47:01.826136 4999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:01.827492 4985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:01.827620 4985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "fd6fd1bb4c484622946f5d36d859de84"
format_stamp: "Formatted at 2026-09-24 04:47:01 on dist-test-slave-hvcl"
server_key: "f9616b2f79f6df03a1de27e3d0753ac5"
server_key_iv: "7972089960ca67f5b03fe4ea96d84091"
server_key_version: "encryptionkey@0"
I20260924 04:47:01.827724 4985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:01.832213 4985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:01.832800 4985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:01.832962 4985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:01.839514 4985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40351
I20260924 04:47:01.839564 5051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40351 every 8 connection(s)
I20260924 04:47:01.840425 4985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:47:01.843580 5052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:01.847544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 4985
I20260924 04:47:01.847661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:47:01.847929 392 external_mini_cluster.cc:1428] Setting key d34b410553dcf5298bf40dc9fa5f10ef
I20260924 04:47:01.849529 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84: Bootstrap starting.
I20260924 04:47:01.851852 5052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:01.852648 5052 log.cc:824] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:01.853673 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:01.849385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.0.98.62:40351)
0924 04:47:01.850101 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:01.850118 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:01.850155 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:47:01.850807 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:01.850825 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:01.851251 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:01.851581 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:01.852382 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:01.852386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:01.852399 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:01.852410 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:01.852471 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:01.852488 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:01.852579 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:01.852705 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 04:47:01.852976 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":135,"threads_started":1}
I20260924 04:47:01.854682 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84: No bootstrap required, opened a new log
I20260924 04:47:01.858455 5052 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6fd1bb4c484622946f5d36d859de84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40351 } }
I20260924 04:47:01.858680 5052 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:01.858736 5052 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd6fd1bb4c484622946f5d36d859de84, State: Initialized, Role: FOLLOWER
I20260924 04:47:01.859175 5052 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6fd1bb4c484622946f5d36d859de84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40351 } }
I20260924 04:47:01.859313 5052 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:47:01.859376 5052 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:47:01.859469 5052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:01.860330 5052 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6fd1bb4c484622946f5d36d859de84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40351 } }
I20260924 04:47:01.860638 5052 leader_election.cc:302] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd6fd1bb4c484622946f5d36d859de84; no voters:
I20260924 04:47:01.860934 5052 leader_election.cc:288] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:47:01.861016 5057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:01.861167 5057 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [term 1 LEADER]: Becoming Leader. State: Replica: fd6fd1bb4c484622946f5d36d859de84, State: Running, Role: LEADER
I20260924 04:47:01.861452 5057 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6fd1bb4c484622946f5d36d859de84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40351 } }
I20260924 04:47:01.862190 5052 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:01.863118 5058 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd6fd1bb4c484622946f5d36d859de84. Latest consensus state: current_term: 1 leader_uuid: "fd6fd1bb4c484622946f5d36d859de84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6fd1bb4c484622946f5d36d859de84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40351 } } }
I20260924 04:47:01.863255 5058 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:01.863530 5059 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd6fd1bb4c484622946f5d36d859de84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6fd1bb4c484622946f5d36d859de84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40351 } } }
I20260924 04:47:01.863619 5059 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:01.863727 5066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:01.866709 5066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:01.872359 5066 catalog_manager.cc:1347] Generated new cluster ID: 357e1a779c9c4e47b35efbe325a1df26
I20260924 04:47:01.872440 5066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:01.896945 5066 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:47:01.897286 5066 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:01.906379 5066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84: Generated new TSK 0
I20260924 04:47:01.907265 5066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:47:01.918766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40351
--builtin_ntp_servers=127.0.98.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:47:02.030035 5076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:02.030316 5076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:02.030365 5076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:02.034704 5076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:02.034767 5076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:02.034827 5076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:47:02.039980 5076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:02.041110 5076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:02.042573 5076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:47:02.052829 5076 server_base.cc:1034] running on GCE node
W20260924 04:47:02.052788 5081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:02.052800 5084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:02.052790 5082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:02.053514 5076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:02.054247 5076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:02.055433 5076 hybrid_clock.cc:648] HybridClock initialized: now 1790225222055418 us; error 28 us; skew 500 ppm
I20260924 04:47:02.057328 5076 webserver.cc:460] Webserver started at http://127.0.98.1:38635/ using document root <none> and password file <none>
I20260924 04:47:02.057888 5076 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:02.058012 5076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:02.058223 5076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:02.060312 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "3d73f26938fc4e05be7c6dceed9f09ea"
format_stamp: "Formatted at 2026-09-24 04:47:02 on dist-test-slave-hvcl"
server_key: "4a0fadefde5a7051fb47fcc747fc6ca7"
server_key_iv: "b3ffbe61415f95fe6af95ac7658624c3"
server_key_version: "encryptionkey@0"
I20260924 04:47:02.060828 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3d73f26938fc4e05be7c6dceed9f09ea"
format_stamp: "Formatted at 2026-09-24 04:47:02 on dist-test-slave-hvcl"
server_key: "4a0fadefde5a7051fb47fcc747fc6ca7"
server_key_iv: "b3ffbe61415f95fe6af95ac7658624c3"
server_key_version: "encryptionkey@0"
I20260924 04:47:02.064613 5076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 04:47:02.067116 5090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:02.068351 5076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:47:02.068457 5076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "3d73f26938fc4e05be7c6dceed9f09ea"
format_stamp: "Formatted at 2026-09-24 04:47:02 on dist-test-slave-hvcl"
server_key: "4a0fadefde5a7051fb47fcc747fc6ca7"
server_key_iv: "b3ffbe61415f95fe6af95ac7658624c3"
server_key_version: "encryptionkey@0"
I20260924 04:47:02.068554 5076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:02.073030 5076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:02.073609 5076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:02.073755 5076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:02.074260 5076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:02.075162 5076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:02.075206 5076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:02.075242 5076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:02.075263 5076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:02.083266 5076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37987
I20260924 04:47:02.083340 5203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37987 every 8 connection(s)
I20260924 04:47:02.084144 5076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:47:02.084616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 5076
I20260924 04:47:02.084743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:47:02.085063 392 external_mini_cluster.cc:1428] Setting key 602587c5f4705a7bd16dd6ed6dd6468d
I20260924 04:47:02.087460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40351
--builtin_ntp_servers=127.0.98.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 04:47:02.090638 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.086116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45631 (local address 127.0.98.62:40351)
0924 04:47:02.086230 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:02.086233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:02.087663 (+ 1430us) server_negotiation.cc:407] Connection header received
0924 04:47:02.088965 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:02.088968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:02.089011 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:02.089093 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:02.090206 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:02.090210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:02.090212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:02.090226 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:02.090247 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:02.090255 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:02.090334 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:02.090452 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 04:47:02.090498 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:47:02.091701 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.086728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40351 (local address 127.0.98.1:45631)
0924 04:47:02.087505 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:02.087570 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:02.088754 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:02.089278 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:02.089285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:02.089379 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:02.089937 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:47:02.089939 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:02.089981 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:47:02.090063 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:02.090323 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:02.090331 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:02.090336 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:47:02.090493 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 04:47:02.090773 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20260924 04:47:02.092954 5204 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40351
I20260924 04:47:02.093182 5204 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:02.093581 5204 heartbeater.cc:511] Master 127.0.98.62:40351 requested a full tablet report, sending...
I20260924 04:47:02.095259 5016 ts_manager.cc:194] Registered new tserver with Master: 3d73f26938fc4e05be7c6dceed9f09ea (127.0.98.1:37987)
W20260924 04:47:02.197119 5207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:02.197379 5207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:02.197425 5207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:02.201354 5207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:02.201411 5207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:02.201467 5207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:47:02.206142 5207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:02.207255 5207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:02.208645 5207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:02.218134 5213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:02.218143 5212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:02.218152 5215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:02.218324 5207 server_base.cc:1034] running on GCE node
I20260924 04:47:02.218925 5207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:02.219643 5207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:02.220804 5207 hybrid_clock.cc:648] HybridClock initialized: now 1790225222220782 us; error 34 us; skew 500 ppm
I20260924 04:47:02.222631 5207 webserver.cc:460] Webserver started at http://127.0.98.2:35079/ using document root <none> and password file <none>
I20260924 04:47:02.223228 5207 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:02.223297 5207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:02.223492 5207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:02.225615 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "5e28b9ce17cf4a318545ad145e6232df"
format_stamp: "Formatted at 2026-09-24 04:47:02 on dist-test-slave-hvcl"
server_key: "5164359bc0da65cf7d7c481877a8abaa"
server_key_iv: "e8837ab52d0c37dbb8589d3ed110e93a"
server_key_version: "encryptionkey@0"
I20260924 04:47:02.226179 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5e28b9ce17cf4a318545ad145e6232df"
format_stamp: "Formatted at 2026-09-24 04:47:02 on dist-test-slave-hvcl"
server_key: "5164359bc0da65cf7d7c481877a8abaa"
server_key_iv: "e8837ab52d0c37dbb8589d3ed110e93a"
server_key_version: "encryptionkey@0"
I20260924 04:47:02.229856 5207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:47:02.232410 5221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:02.233675 5207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:47:02.233793 5207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "5e28b9ce17cf4a318545ad145e6232df"
format_stamp: "Formatted at 2026-09-24 04:47:02 on dist-test-slave-hvcl"
server_key: "5164359bc0da65cf7d7c481877a8abaa"
server_key_iv: "e8837ab52d0c37dbb8589d3ed110e93a"
server_key_version: "encryptionkey@0"
I20260924 04:47:02.233927 5207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:02.238463 5207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:02.239089 5207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:02.239264 5207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:02.239893 5207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:02.240775 5207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:02.240819 5207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:02.240870 5207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:02.240898 5207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:02.249382 5207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42137
I20260924 04:47:02.249411 5334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42137 every 8 connection(s)
I20260924 04:47:02.250274 5207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:47:02.254640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 5207
I20260924 04:47:02.254778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:47:02.255107 392 external_mini_cluster.cc:1428] Setting key 7b4e1fb1eaf04fe5575662325d828180
I20260924 04:47:02.256946 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.252149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33637 (local address 127.0.98.62:40351)
0924 04:47:02.252297 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:02.252301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:02.253446 (+ 1145us) server_negotiation.cc:407] Connection header received
0924 04:47:02.254885 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:02.254888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:02.254928 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:02.255048 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:02.256315 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:02.256318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:02.256321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:02.256333 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:02.256355 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:02.256364 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:02.256453 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:02.256756 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 04:47:02.256804 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:47:02.257093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40351
--builtin_ntp_servers=127.0.98.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 04:47:02.257893 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.252546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40351 (local address 127.0.98.2:33637)
0924 04:47:02.253284 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:02.253325 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:02.254649 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:02.255234 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:02.255244 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:02.255363 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:02.256056 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:47:02.256058 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:02.256098 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:47:02.256167 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:02.256451 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:02.256458 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:02.256462 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:47:02.256805 (+ 343us) client_negotiation.cc:240] Negotiation successful
0924 04:47:02.257058 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":108,"threads_started":1}
I20260924 04:47:02.259169 5335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40351
I20260924 04:47:02.259423 5335 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:02.259843 5335 heartbeater.cc:511] Master 127.0.98.62:40351 requested a full tablet report, sending...
I20260924 04:47:02.260890 5016 ts_manager.cc:194] Registered new tserver with Master: 5e28b9ce17cf4a318545ad145e6232df (127.0.98.2:42137)
W20260924 04:47:02.367093 5338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:02.367372 5338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:02.367412 5338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:02.371519 5338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:02.371573 5338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:02.371631 5338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:47:02.376444 5338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:02.377496 5338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:02.378824 5338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:02.388331 5344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:02.388331 5346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:02.388387 5343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:02.388643 5338 server_base.cc:1034] running on GCE node
I20260924 04:47:02.389045 5338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:02.389673 5338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:02.390837 5338 hybrid_clock.cc:648] HybridClock initialized: now 1790225222390813 us; error 40 us; skew 500 ppm
I20260924 04:47:02.392710 5338 webserver.cc:460] Webserver started at http://127.0.98.3:42851/ using document root <none> and password file <none>
I20260924 04:47:02.393309 5338 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:02.393373 5338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:02.393558 5338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:02.395610 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "7c6d0589a8e242ec9657cd97a537301d"
format_stamp: "Formatted at 2026-09-24 04:47:02 on dist-test-slave-hvcl"
server_key: "b6ed176971da98c0b5ee84415e2f27d6"
server_key_iv: "0ad42e9d981024449a7e6e05fd0b7263"
server_key_version: "encryptionkey@0"
I20260924 04:47:02.396126 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7c6d0589a8e242ec9657cd97a537301d"
format_stamp: "Formatted at 2026-09-24 04:47:02 on dist-test-slave-hvcl"
server_key: "b6ed176971da98c0b5ee84415e2f27d6"
server_key_iv: "0ad42e9d981024449a7e6e05fd0b7263"
server_key_version: "encryptionkey@0"
I20260924 04:47:02.399690 5338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:47:02.402034 5352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:02.403385 5338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:02.403496 5338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "7c6d0589a8e242ec9657cd97a537301d"
format_stamp: "Formatted at 2026-09-24 04:47:02 on dist-test-slave-hvcl"
server_key: "b6ed176971da98c0b5ee84415e2f27d6"
server_key_iv: "0ad42e9d981024449a7e6e05fd0b7263"
server_key_version: "encryptionkey@0"
I20260924 04:47:02.403594 5338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:02.408339 5338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:02.408950 5338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:02.409111 5338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:02.409668 5338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:02.410578 5338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:02.410625 5338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:02.410676 5338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:02.410702 5338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:02.420221 5338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40003
I20260924 04:47:02.420230 5465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40003 every 8 connection(s)
I20260924 04:47:02.421162 5338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:47:02.423614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 5338
I20260924 04:47:02.423718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:47:02.423979 392 external_mini_cluster.cc:1428] Setting key 9cc73d435bf0b2ea9fc4ae6b74050dfc
I20260924 04:47:02.427421 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.423131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52655 (local address 127.0.98.62:40351)
0924 04:47:02.423266 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:02.423270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:02.424360 (+ 1090us) server_negotiation.cc:407] Connection header received
0924 04:47:02.425824 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:02.425827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:02.425869 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:02.425945 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:02.427006 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:02.427011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:02.427013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:02.427027 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:02.427045 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:02.427054 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:02.427137 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:02.427278 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 04:47:02.427317 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:47:02.428251 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.423475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40351 (local address 127.0.98.3:52655)
0924 04:47:02.424206 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:02.424259 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:02.425626 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:02.426122 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:02.426129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:02.426224 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:02.426729 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:47:02.426731 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:02.426772 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:47:02.426845 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:02.427130 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:02.427135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:02.427139 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:47:02.427271 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 04:47:02.427474 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20260924 04:47:02.429270 5466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40351
I20260924 04:47:02.429512 5466 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:02.429910 5466 heartbeater.cc:511] Master 127.0.98.62:40351 requested a full tablet report, sending...
I20260924 04:47:02.430920 5016 ts_manager.cc:194] Registered new tserver with Master: 7c6d0589a8e242ec9657cd97a537301d (127.0.98.3:40003)
I20260924 04:47:02.438237 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:47:02.441874 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.440265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.0.98.62:40351)
0924 04:47:02.440408 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:02.440413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:02.440547 (+ 134us) server_negotiation.cc:407] Connection header received
0924 04:47:02.440681 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:02.440684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:02.440723 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:02.440787 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:02.441485 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:02.441488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:02.441490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:02.441500 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:02.441519 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:02.441526 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:02.441616 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:02.441681 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 04:47:02.441724 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:47:02.445137 5016 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:47:02.448154 5016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:47:02.459806 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.456667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40003 (local address 127.0.0.1:46604)
0924 04:47:02.457083 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:02.457128 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:02.457244 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:02.458411 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:02.458417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:02.458495 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:02.459075 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:47:02.459077 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:02.459098 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:47:02.459177 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:02.459550 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:02.459557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:02.459563 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:47:02.459640 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:47:02.459694 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":83,"threads_started":1}
I20260924 04:47:02.460106 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.457572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.0.98.3:40003)
0924 04:47:02.458023 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:02.458033 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:02.458052 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:47:02.458098 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:02.458102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:02.458242 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:02.458790 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:02.459385 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:02.459391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:02.459404 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:02.459414 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:02.459453 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:02.459478 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:02.459584 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:02.459658 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:47:02.459730 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":126,"threads_started":1}
I20260924 04:47:02.460414 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.457366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37987 (local address 127.0.0.1:50160)
0924 04:47:02.457773 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:02.457787 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:02.457862 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:02.458637 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:02.458640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:02.458655 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:02.459305 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:47:02.459306 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:02.459316 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:47:02.459349 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:02.460161 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:02.460164 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:02.460168 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:47:02.460260 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 04:47:02.460295 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":96,"threads_started":1}
I20260924 04:47:02.460500 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.457557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50160 (local address 127.0.98.1:37987)
0924 04:47:02.458214 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:02.458232 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:02.458255 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:47:02.458305 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:02.458310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:02.458490 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:02.458620 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:02.459778 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:02.459784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:02.459794 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:02.459803 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:02.459838 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:02.459870 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:02.460180 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:02.460275 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 04:47:02.460347 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":130,"threads_started":1}
I20260924 04:47:02.460875 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.456914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34828 (local address 127.0.98.2:42137)
0924 04:47:02.457485 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:02.457497 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:02.457724 (+ 227us) server_negotiation.cc:407] Connection header received
0924 04:47:02.457910 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:02.457916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:02.458118 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:02.458224 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:02.460133 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:02.460139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:02.460151 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:02.460163 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:02.460206 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:02.460239 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:02.460328 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:02.460647 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 04:47:02.460724 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":78,"threads_started":1}
I20260924 04:47:02.461143 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.456746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42137 (local address 127.0.0.1:34828)
0924 04:47:02.457630 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:02.457643 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:02.457783 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:02.458944 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:02.458947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:02.458961 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:02.459981 (+ 1020us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:47:02.459983 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:02.459993 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:47:02.460018 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:02.460366 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:02.460369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:02.460371 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 04:47:02.461005 (+ 634us) client_negotiation.cc:240] Negotiation successful
0924 04:47:02.461039 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":117,"threads_started":1}
I20260924 04:47:02.463349 5138 tablet_service.cc:1467] Processing CreateTablet for tablet a266eeaff44f461ea34ffa1d28a833b9 (DEFAULT_TABLE table=test-table [id=27175a35e7e84082b0e7e969ca92d85c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:02.463379 5399 tablet_service.cc:1467] Processing CreateTablet for tablet a266eeaff44f461ea34ffa1d28a833b9 (DEFAULT_TABLE table=test-table [id=27175a35e7e84082b0e7e969ca92d85c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:02.463611 5269 tablet_service.cc:1467] Processing CreateTablet for tablet a266eeaff44f461ea34ffa1d28a833b9 (DEFAULT_TABLE table=test-table [id=27175a35e7e84082b0e7e969ca92d85c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:02.464490 5269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a266eeaff44f461ea34ffa1d28a833b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:02.464490 5138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a266eeaff44f461ea34ffa1d28a833b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:02.464490 5399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a266eeaff44f461ea34ffa1d28a833b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:02.470844 5483 tablet_bootstrap.cc:492] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df: Bootstrap starting.
I20260924 04:47:02.472469 5484 tablet_bootstrap.cc:492] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea: Bootstrap starting.
I20260924 04:47:02.473093 5483 tablet_bootstrap.cc:654] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:02.473812 5483 log.cc:824] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:02.474510 5484 tablet_bootstrap.cc:654] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:02.475196 5484 log.cc:824] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:02.475311 5485 tablet_bootstrap.cc:492] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d: Bootstrap starting.
I20260924 04:47:02.475749 5483 tablet_bootstrap.cc:492] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df: No bootstrap required, opened a new log
I20260924 04:47:02.475951 5483 ts_tablet_manager.cc:1397] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:47:02.476764 5484 tablet_bootstrap.cc:492] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea: No bootstrap required, opened a new log
I20260924 04:47:02.476938 5484 ts_tablet_manager.cc:1397] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:47:02.477377 5485 tablet_bootstrap.cc:654] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:02.478155 5485 log.cc:824] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:02.480327 5485 tablet_bootstrap.cc:492] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d: No bootstrap required, opened a new log
I20260924 04:47:02.480175 5483 raft_consensus.cc:348] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6d0589a8e242ec9657cd97a537301d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40003 } } peers { permanent_uuid: "5e28b9ce17cf4a318545ad145e6232df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42137 } } peers { permanent_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37987 } }
I20260924 04:47:02.480435 5483 raft_consensus.cc:374] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:02.480489 5483 raft_consensus.cc:729] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e28b9ce17cf4a318545ad145e6232df, State: Initialized, Role: FOLLOWER
I20260924 04:47:02.480515 5485 ts_tablet_manager.cc:1397] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:47:02.480523 5484 raft_consensus.cc:348] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6d0589a8e242ec9657cd97a537301d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40003 } } peers { permanent_uuid: "5e28b9ce17cf4a318545ad145e6232df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42137 } } peers { permanent_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37987 } }
I20260924 04:47:02.480757 5484 raft_consensus.cc:374] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:02.480809 5484 raft_consensus.cc:729] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d73f26938fc4e05be7c6dceed9f09ea, State: Initialized, Role: FOLLOWER
I20260924 04:47:02.480962 5483 consensus_queue.cc:260] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6d0589a8e242ec9657cd97a537301d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40003 } } peers { permanent_uuid: "5e28b9ce17cf4a318545ad145e6232df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42137 } } peers { permanent_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37987 } }
I20260924 04:47:02.481211 5484 consensus_queue.cc:260] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6d0589a8e242ec9657cd97a537301d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40003 } } peers { permanent_uuid: "5e28b9ce17cf4a318545ad145e6232df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42137 } } peers { permanent_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37987 } }
I20260924 04:47:02.481664 5335 heartbeater.cc:503] Master 127.0.98.62:40351 was elected leader, sending a full tablet report...
I20260924 04:47:02.481941 5483 ts_tablet_manager.cc:1428] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260924 04:47:02.482070 5484 ts_tablet_manager.cc:1428] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:47:02.482381 5204 heartbeater.cc:503] Master 127.0.98.62:40351 was elected leader, sending a full tablet report...
I20260924 04:47:02.484941 5485 raft_consensus.cc:348] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6d0589a8e242ec9657cd97a537301d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40003 } } peers { permanent_uuid: "5e28b9ce17cf4a318545ad145e6232df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42137 } } peers { permanent_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37987 } }
I20260924 04:47:02.485221 5485 raft_consensus.cc:374] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:02.485280 5485 raft_consensus.cc:729] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6d0589a8e242ec9657cd97a537301d, State: Initialized, Role: FOLLOWER
I20260924 04:47:02.485702 5485 consensus_queue.cc:260] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6d0589a8e242ec9657cd97a537301d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40003 } } peers { permanent_uuid: "5e28b9ce17cf4a318545ad145e6232df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42137 } } peers { permanent_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37987 } }
I20260924 04:47:02.486552 5466 heartbeater.cc:503] Master 127.0.98.62:40351 was elected leader, sending a full tablet report...
I20260924 04:47:02.486655 5485 ts_tablet_manager.cc:1428] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:47:02.492480 5490 raft_consensus.cc:482] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:47:02.492659 5490 raft_consensus.cc:504] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6d0589a8e242ec9657cd97a537301d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40003 } } peers { permanent_uuid: "5e28b9ce17cf4a318545ad145e6232df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42137 } } peers { permanent_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37987 } }
I20260924 04:47:02.493690 5490 leader_election.cc:288] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c6d0589a8e242ec9657cd97a537301d (127.0.98.3:40003), 5e28b9ce17cf4a318545ad145e6232df (127.0.98.2:42137)
I20260924 04:47:02.495628 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.494034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43515 (local address 127.0.98.3:40003)
0924 04:47:02.494202 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:02.494207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:02.494221 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:47:02.494334 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:02.494336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:02.494378 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:02.494448 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:02.495210 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:02.495213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:02.495216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:02.495230 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:02.495252 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:02.495262 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:02.495345 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:02.495480 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 04:47:02.495522 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260924 04:47:02.495640 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.493949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40003 (local address 127.0.98.1:43515)
0924 04:47:02.494137 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:02.494170 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:02.494245 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:02.494531 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:02.494534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:02.494549 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:02.495072 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:47:02.495074 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:02.495087 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:47:02.495115 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:02.495359 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:02.495363 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:02.495368 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:47:02.495489 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 04:47:02.495525 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 04:47:02.496084 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.494075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44025 (local address 127.0.98.2:42137)
0924 04:47:02.494177 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:02.494181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:02.494445 (+ 264us) server_negotiation.cc:407] Connection header received
0924 04:47:02.494592 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:02.494595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:02.494633 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:02.494714 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:02.495652 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:02.495656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:02.495659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:02.495671 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:02.495692 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:02.495702 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:02.495784 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:02.495932 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 04:47:02.495975 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 04:47:02.496140 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.493949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42137 (local address 127.0.98.1:44025)
0924 04:47:02.494332 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:02.494346 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:02.494447 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:02.495068 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:02.495071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:02.495086 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:02.495502 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:47:02.495504 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:02.495514 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:47:02.495538 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:02.495824 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:02.495828 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:02.495831 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:47:02.495969 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 04:47:02.496004 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":31,"thread_start_us":120,"threads_started":1}
I20260924 04:47:02.496304 5407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a266eeaff44f461ea34ffa1d28a833b9" candidate_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6d0589a8e242ec9657cd97a537301d" is_pre_election: true
I20260924 04:47:02.496565 5407 raft_consensus.cc:2454] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d73f26938fc4e05be7c6dceed9f09ea in term 0.
I20260924 04:47:02.496986 5094 leader_election.cc:302] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d73f26938fc4e05be7c6dceed9f09ea, 7c6d0589a8e242ec9657cd97a537301d; no voters:
I20260924 04:47:02.497237 5490 raft_consensus.cc:2781] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:47:02.497334 5490 raft_consensus.cc:482] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:47:02.497361 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a266eeaff44f461ea34ffa1d28a833b9" candidate_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e28b9ce17cf4a318545ad145e6232df" is_pre_election: true
I20260924 04:47:02.497432 5490 raft_consensus.cc:3037] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:02.497630 5289 raft_consensus.cc:2454] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d73f26938fc4e05be7c6dceed9f09ea in term 0.
I20260924 04:47:02.498628 5490 raft_consensus.cc:504] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6d0589a8e242ec9657cd97a537301d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40003 } } peers { permanent_uuid: "5e28b9ce17cf4a318545ad145e6232df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42137 } } peers { permanent_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37987 } }
I20260924 04:47:02.498935 5490 leader_election.cc:288] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [CANDIDATE]: Term 1 election: Requested vote from peers 7c6d0589a8e242ec9657cd97a537301d (127.0.98.3:40003), 5e28b9ce17cf4a318545ad145e6232df (127.0.98.2:42137)
I20260924 04:47:02.499253 5407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a266eeaff44f461ea34ffa1d28a833b9" candidate_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6d0589a8e242ec9657cd97a537301d"
I20260924 04:47:02.499275 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a266eeaff44f461ea34ffa1d28a833b9" candidate_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e28b9ce17cf4a318545ad145e6232df"
I20260924 04:47:02.499357 5407 raft_consensus.cc:3037] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:02.499387 5289 raft_consensus.cc:3037] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:02.500326 5407 raft_consensus.cc:2454] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d73f26938fc4e05be7c6dceed9f09ea in term 1.
I20260924 04:47:02.500479 5289 raft_consensus.cc:2454] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d73f26938fc4e05be7c6dceed9f09ea in term 1.
I20260924 04:47:02.500576 5094 leader_election.cc:302] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d73f26938fc4e05be7c6dceed9f09ea, 7c6d0589a8e242ec9657cd97a537301d; no voters:
I20260924 04:47:02.500741 5490 raft_consensus.cc:2781] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:02.500957 5490 raft_consensus.cc:686] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [term 1 LEADER]: Becoming Leader. State: Replica: 3d73f26938fc4e05be7c6dceed9f09ea, State: Running, Role: LEADER
W20260924 04:47:02.501118 5336 tablet.cc:2369] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:47:02.501293 5490 consensus_queue.cc:237] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6d0589a8e242ec9657cd97a537301d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40003 } } peers { permanent_uuid: "5e28b9ce17cf4a318545ad145e6232df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42137 } } peers { permanent_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37987 } }
I20260924 04:47:02.504324 5016 catalog_manager.cc:5495] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d73f26938fc4e05be7c6dceed9f09ea (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6d0589a8e242ec9657cd97a537301d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e28b9ce17cf4a318545ad145e6232df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d73f26938fc4e05be7c6dceed9f09ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37987 } health_report { overall_health: HEALTHY } } }
I20260924 04:47:02.524142 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.522653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50170 (local address 127.0.98.1:37987)
0924 04:47:02.522767 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:02.522771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:02.522784 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:47:02.522885 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:02.522888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:02.522948 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:02.523070 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:02.523828 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:02.523831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:02.523833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:02.523845 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:02.523866 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:02.523875 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:02.523952 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:02.523994 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 04:47:02.524026 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:47:02.530108 5407 raft_consensus.cc:1261] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d [term 1 FOLLOWER]: Refusing update from remote peer 3d73f26938fc4e05be7c6dceed9f09ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:02.530108 5289 raft_consensus.cc:1261] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df [term 1 FOLLOWER]: Refusing update from remote peer 3d73f26938fc4e05be7c6dceed9f09ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:02.530854 5490 consensus_queue.cc:1035] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e28b9ce17cf4a318545ad145e6232df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:02.531114 5493 consensus_queue.cc:1035] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6d0589a8e242ec9657cd97a537301d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:02.539341 5498 mvcc.cc:205] Tried to move back new op lower bound from 7332762511475642368 to 7332762511367757824. Current Snapshot: MvccSnapshot[applied={T|T < 7332762511475642368}]
I20260924 04:47:02.539554 5497 mvcc.cc:205] Tried to move back new op lower bound from 7332762511475642368 to 7332762511367757824. Current Snapshot: MvccSnapshot[applied={T|T < 7332762511475642368}]
I20260924 04:47:02.545980 5010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:02.546141 5010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:02.548454 5010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd6fd1bb4c484622946f5d36d859de84: Sending DeleteTablet for 3 replicas of tablet a266eeaff44f461ea34ffa1d28a833b9
I20260924 04:47:02.549223 5269 tablet_service.cc:1514] Processing DeleteTablet for tablet a266eeaff44f461ea34ffa1d28a833b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:02 UTC) from {username='slave'} at 127.0.0.1:34828
I20260924 04:47:02.549229 5399 tablet_service.cc:1514] Processing DeleteTablet for tablet a266eeaff44f461ea34ffa1d28a833b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:02 UTC) from {username='slave'} at 127.0.0.1:46604
I20260924 04:47:02.549568 5138 tablet_service.cc:1514] Processing DeleteTablet for tablet a266eeaff44f461ea34ffa1d28a833b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:02 UTC) from {username='slave'} at 127.0.0.1:50160
I20260924 04:47:02.549850 5505 tablet_replica.cc:330] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df: stopping tablet replica
I20260924 04:47:02.549985 5507 tablet_replica.cc:330] T a266eeaff44f461ea34ffa1d28a833b9 P 3d73f26938fc4e05be7c6dceed9f09ea: stopping tablet replica
I20260924 04:47:02.550037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 5076
I20260924 04:47:02.551843 5505 raft_consensus.cc:2229] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:02.552153 5505 raft_consensus.cc:2258] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:02.550017 5506 tablet_replica.cc:330] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d: stopping tablet replica
I20260924 04:47:02.552615 5506 raft_consensus.cc:2229] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:02.552896 5506 raft_consensus.cc:2258] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:02.553741 5505 ts_tablet_manager.cc:1905] T a266eeaff44f461ea34ffa1d28a833b9 P 5e28b9ce17cf4a318545ad145e6232df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:02.554334 5506 ts_tablet_manager.cc:1905] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:47:02.555452 5002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37987 (error 108)
I20260924 04:47:02.555665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 5207
I20260924 04:47:02.556721 5506 ts_tablet_manager.cc:1918] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:02.556795 5506 log.cc:1196] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/a266eeaff44f461ea34ffa1d28a833b9
I20260924 04:47:02.557039 5506 ts_tablet_manager.cc:1939] T a266eeaff44f461ea34ffa1d28a833b9 P 7c6d0589a8e242ec9657cd97a537301d: Deleting consensus metadata
I20260924 04:47:02.558211 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.557916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37987 (local address 127.0.0.1:50182)
0924 04:47:02.558047 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:02.558125 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:47:02.558391 5002 catalog_manager.cc:4584] TS 3d73f26938fc4e05be7c6dceed9f09ea (127.0.98.1:37987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a266eeaff44f461ea34ffa1d28a833b9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37987: connect: Connection refused (error 111)
I20260924 04:47:02.559059 5003 catalog_manager.cc:4841] TS 7c6d0589a8e242ec9657cd97a537301d (127.0.98.3:40003): tablet a266eeaff44f461ea34ffa1d28a833b9 (table test-table [id=27175a35e7e84082b0e7e969ca92d85c]) successfully deleted
I20260924 04:47:02.561239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 5338
I20260924 04:47:02.562618 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.561352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42137 (local address 127.0.0.1:34832)
0924 04:47:02.562477 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:02.562533 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1050}
I20260924 04:47:02.565991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 4985
2026-09-24T04:47:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (911 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 04:47:02.586078 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:47:02.587773 392 webserver.cc:460] Webserver started at https://127.0.0.1:36745/ using document root <none> and password file <none>
I20260924 04:47:02.587929 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:47:02.588182 392 webserver.cc:460] Webserver started at http://127.0.0.1:43683/ using document root <none> and password file <none>
2026-09-24T04:47:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:02Z Disabled control of system clock
I20260924 04:47:02.596333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42937
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42937
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36745/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/testchainca.cert with env {}
W20260924 04:47:02.703697 5514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:02.703970 5514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:02.704008 5514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:02.708034 5514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:02.708086 5514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:02.708099 5514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:02.708115 5514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:02.708128 5514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:47:02.708142 5514 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:47:02.708154 5514 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36745/jwks.json
I20260924 04:47:02.713747 5514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36745/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:02.714742 5514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:02.715752 5514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:02.723738 5522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:02.723749 5520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:02.723851 5514 server_base.cc:1034] running on GCE node
W20260924 04:47:02.723786 5519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:02.724509 5514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:02.725586 5514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:02.726764 5514 hybrid_clock.cc:648] HybridClock initialized: now 1790225222726734 us; error 40 us; skew 500 ppm
I20260924 04:47:02.728658 5514 webserver.cc:460] Webserver started at http://127.0.98.62:37259/ using document root <none> and password file <none>
I20260924 04:47:02.729280 5514 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:02.729347 5514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:02.729525 5514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:02.731587 5514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "9b0bce113b514229b2aaab1c315909d4"
format_stamp: "Formatted at 2026-09-24 04:47:02 on dist-test-slave-hvcl"
server_key: "c6068e7be1912f52fb9952ac6212349e"
server_key_iv: "3ea726e7c5e25fc2f66a3ef5da7018a2"
server_key_version: "encryptionkey@0"
I20260924 04:47:02.732131 5514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b0bce113b514229b2aaab1c315909d4"
format_stamp: "Formatted at 2026-09-24 04:47:02 on dist-test-slave-hvcl"
server_key: "c6068e7be1912f52fb9952ac6212349e"
server_key_iv: "3ea726e7c5e25fc2f66a3ef5da7018a2"
server_key_version: "encryptionkey@0"
I20260924 04:47:02.735682 5514 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:47:02.738039 5528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:02.739462 5514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:02.739578 5514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "9b0bce113b514229b2aaab1c315909d4"
format_stamp: "Formatted at 2026-09-24 04:47:02 on dist-test-slave-hvcl"
server_key: "c6068e7be1912f52fb9952ac6212349e"
server_key_iv: "3ea726e7c5e25fc2f66a3ef5da7018a2"
server_key_version: "encryptionkey@0"
I20260924 04:47:02.739672 5514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:02.791795 5514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:02.792546 5514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:02.792730 5514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:02.799368 5514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42937
I20260924 04:47:02.799389 5580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42937 every 8 connection(s)
I20260924 04:47:02.800312 5514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:47:02.802095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 5514
I20260924 04:47:02.802214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:47:02.802471 392 external_mini_cluster.cc:1428] Setting key ec2ca451cbbb0578d1b3788648381eb4
I20260924 04:47:02.804039 5581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:02.809720 5581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4: Bootstrap starting.
I20260924 04:47:02.811970 5581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:02.811990 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.804078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.0.98.62:42937)
0924 04:47:02.804785 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:02.804801 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:02.804839 (+ 38us) server_negotiation.cc:407] Connection header received
0924 04:47:02.805517 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:02.805533 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:02.805941 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:02.806267 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:02.806724 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:02.807758 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:02.808643 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:02.809124 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:02.809272 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:02.810216 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:02.810221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:02.810234 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:02.810642 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:02.810700 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:02.810721 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:02.810804 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:02.810954 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 04:47:02.811254 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":139,"threads_started":1}
I20260924 04:47:02.812685 5581 log.cc:824] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:02.814711 5581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4: No bootstrap required, opened a new log
I20260924 04:47:02.814759 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:47:02.818562 5581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0bce113b514229b2aaab1c315909d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42937 } }
I20260924 04:47:02.818782 5581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:02.818842 5581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0bce113b514229b2aaab1c315909d4, State: Initialized, Role: FOLLOWER
I20260924 04:47:02.819257 5581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0bce113b514229b2aaab1c315909d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42937 } }
I20260924 04:47:02.819380 5581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:47:02.819440 5581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:47:02.819532 5581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:02.820375 5581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0bce113b514229b2aaab1c315909d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42937 } }
I20260924 04:47:02.820683 5581 leader_election.cc:302] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b0bce113b514229b2aaab1c315909d4; no voters:
I20260924 04:47:02.820982 5581 leader_election.cc:288] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:47:02.821184 5586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:02.821444 5586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [term 1 LEADER]: Becoming Leader. State: Replica: 9b0bce113b514229b2aaab1c315909d4, State: Running, Role: LEADER
I20260924 04:47:02.821729 5586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0bce113b514229b2aaab1c315909d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42937 } }
I20260924 04:47:02.822156 5581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:02.823302 5588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b0bce113b514229b2aaab1c315909d4. Latest consensus state: current_term: 1 leader_uuid: "9b0bce113b514229b2aaab1c315909d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0bce113b514229b2aaab1c315909d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42937 } } }
I20260924 04:47:02.823335 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b0bce113b514229b2aaab1c315909d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0bce113b514229b2aaab1c315909d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42937 } } }
I20260924 04:47:02.823432 5588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:02.823438 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:02.823863 5596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:02.826996 5596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:02.832614 5596 catalog_manager.cc:1347] Generated new cluster ID: 62039451fdcb4c63be59024dd5bdb42a
I20260924 04:47:02.832705 5596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:02.844573 5596 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:47:02.845535 5596 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:02.857607 5596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b0bce113b514229b2aaab1c315909d4: Generated new TSK 0
I20260924 04:47:02.858315 5596 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 04:47:02.886729 5611 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 04:47:02.886833 5611 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 04:47:02.887542 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:02.870423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54692 (local address 127.0.98.62:42937)
0924 04:47:02.870549 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:02.870554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:02.870725 (+ 171us) server_negotiation.cc:407] Connection header received
0924 04:47:02.870912 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:02.870915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:02.870995 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:02.871115 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:47:02.871320 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:02.872140 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:02.872957 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:02.873313 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:02.873408 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:02.873719 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:47:02.887097 (+ 13378us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 04:47:02.887390 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54692: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 04:47:02.887637 5584 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54692: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 04:47:02.888693 5610 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:02.870319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42937 (local address 127.0.0.1:54692)
0924 04:47:02.870627 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:02.870639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:02.870769 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:02.871117 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:02.871119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:02.871126 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:47:02.871231 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:02.871235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:02.872229 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:02.872231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:02.872860 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:02.872867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:02.873413 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:02.873432 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:02.873507 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:02.873551 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:47:02.887338 (+ 13787us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 04:47:02.887543 (+ 205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42937: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":98,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:42937: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36745/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 04:47:02.889304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 5514
2026-09-24T04:47:02Z chronyd exiting
I20260924 04:47:02.925830 392 test_util.cc:175] -----------------------------------------------
I20260924 04:47:02.925921 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225194677594-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (341 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 04:47:02.927026 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:47:02.927867 392 webserver.cc:460] Webserver started at https://127.0.0.1:45529/ using document root <none> and password file <none>
I20260924 04:47:02.927958 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:47:02.928213 392 webserver.cc:460] Webserver started at http://127.0.0.1:40963/ using document root <none> and password file <none>
2026-09-24T04:47:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:02Z Disabled control of system clock
I20260924 04:47:02.936501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33145
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33145
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45529/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/testchainca.cert with env {}
W20260924 04:47:03.044294 5617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:03.044565 5617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:03.044624 5617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:03.049009 5617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:03.049086 5617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:03.049100 5617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:03.049118 5617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:03.049132 5617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:47:03.049147 5617 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:47:03.049160 5617 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45529/jwks.json
I20260924 04:47:03.054678 5617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33145
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45529/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:03.055804 5617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:03.056819 5617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:03.065083 5623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:03.065079 5625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:03.065079 5622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:03.065752 5617 server_base.cc:1034] running on GCE node
I20260924 04:47:03.066301 5617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:03.067507 5617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:03.068696 5617 hybrid_clock.cc:648] HybridClock initialized: now 1790225223068666 us; error 46 us; skew 500 ppm
I20260924 04:47:03.070614 5617 webserver.cc:460] Webserver started at http://127.0.98.62:43225/ using document root <none> and password file <none>
I20260924 04:47:03.071230 5617 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:03.071308 5617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:03.071507 5617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:03.073510 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "e5c19dcfe2b2419fa89b38ea6b4edb15"
format_stamp: "Formatted at 2026-09-24 04:47:03 on dist-test-slave-hvcl"
server_key: "25fece0180fd9232a6a18e0830a364f2"
server_key_iv: "c9efbbfad6e6b1a61836d34f6ab4826d"
server_key_version: "encryptionkey@0"
I20260924 04:47:03.074024 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "e5c19dcfe2b2419fa89b38ea6b4edb15"
format_stamp: "Formatted at 2026-09-24 04:47:03 on dist-test-slave-hvcl"
server_key: "25fece0180fd9232a6a18e0830a364f2"
server_key_iv: "c9efbbfad6e6b1a61836d34f6ab4826d"
server_key_version: "encryptionkey@0"
I20260924 04:47:03.077667 5617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 04:47:03.080139 5631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:03.081396 5617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:47:03.081514 5617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "e5c19dcfe2b2419fa89b38ea6b4edb15"
format_stamp: "Formatted at 2026-09-24 04:47:03 on dist-test-slave-hvcl"
server_key: "25fece0180fd9232a6a18e0830a364f2"
server_key_iv: "c9efbbfad6e6b1a61836d34f6ab4826d"
server_key_version: "encryptionkey@0"
I20260924 04:47:03.081604 5617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:03.097862 5617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:03.098477 5617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:03.098618 5617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:03.105106 5617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33145
I20260924 04:47:03.105162 5683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33145 every 8 connection(s)
I20260924 04:47:03.105994 5617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:47:03.108948 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:03.112304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 5617
I20260924 04:47:03.112401 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:47:03.112684 392 external_mini_cluster.cc:1428] Setting key 0fd4e42baad7b8188c8ba4221a894ed8
I20260924 04:47:03.114912 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15: Bootstrap starting.
I20260924 04:47:03.117188 5684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:03.117964 5684 log.cc:824] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:03.120065 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15: No bootstrap required, opened a new log
I20260924 04:47:03.121754 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:03.114108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.0.98.62:33145)
0924 04:47:03.114715 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:03.114731 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:03.114767 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:47:03.115471 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:03.115494 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:03.115901 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:03.116186 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:03.116543 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.117538 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.118637 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.119162 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.119306 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:03.120113 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:03.120120 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:03.120137 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:03.120528 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:03.120591 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:03.120610 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:03.120691 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:03.120826 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 04:47:03.121068 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260924 04:47:03.123835 5684 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c19dcfe2b2419fa89b38ea6b4edb15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } }
I20260924 04:47:03.124095 5684 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:03.124166 5684 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c19dcfe2b2419fa89b38ea6b4edb15, State: Initialized, Role: FOLLOWER
I20260924 04:47:03.124584 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:47:03.124586 5684 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c19dcfe2b2419fa89b38ea6b4edb15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } }
I20260924 04:47:03.124716 5684 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:47:03.124785 5684 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:47:03.124879 5684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:03.125808 5684 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c19dcfe2b2419fa89b38ea6b4edb15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } }
I20260924 04:47:03.126149 5684 leader_election.cc:302] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5c19dcfe2b2419fa89b38ea6b4edb15; no voters:
I20260924 04:47:03.126435 5684 leader_election.cc:288] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:47:03.126515 5689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:03.126722 5689 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [term 1 LEADER]: Becoming Leader. State: Replica: e5c19dcfe2b2419fa89b38ea6b4edb15, State: Running, Role: LEADER
I20260924 04:47:03.127028 5689 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c19dcfe2b2419fa89b38ea6b4edb15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } }
I20260924 04:47:03.127714 5684 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:03.128672 5691 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5c19dcfe2b2419fa89b38ea6b4edb15. Latest consensus state: current_term: 1 leader_uuid: "e5c19dcfe2b2419fa89b38ea6b4edb15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c19dcfe2b2419fa89b38ea6b4edb15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } } }
I20260924 04:47:03.128791 5691 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:03.129112 5690 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5c19dcfe2b2419fa89b38ea6b4edb15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c19dcfe2b2419fa89b38ea6b4edb15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } } }
I20260924 04:47:03.129240 5690 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:03.129168 5698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:03.132375 5698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:03.137919 5698 catalog_manager.cc:1347] Generated new cluster ID: 89ec8bc720ec47dd8f2185cb10a98d11
I20260924 04:47:03.137992 5698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:03.149717 5698 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:47:03.150660 5698 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:03.157634 5698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5c19dcfe2b2419fa89b38ea6b4edb15: Generated new TSK 0
I20260924 04:47:03.158358 5698 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 04:47:03.190301 5714 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 04:47:03.190407 5714 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 04:47:03.191102 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:03.172397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.0.98.62:33145)
0924 04:47:03.172523 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:03.172527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:03.172646 (+ 119us) server_negotiation.cc:407] Connection header received
0924 04:47:03.172813 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:03.172816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:03.172863 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:03.172949 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:47:03.173214 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.173734 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.174941 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.175222 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.175302 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:03.175631 (+ 329us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:47:03.190696 (+ 15065us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 04:47:03.190932 (+ 236us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45612: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 04:47:03.191164 5687 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45612: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 04:47:03.191319 5713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:03.172270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33145 (local address 127.0.0.1:45612)
0924 04:47:03.172563 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:03.172577 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:03.172698 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:03.172959 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:03.172961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:03.172969 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:47:03.173105 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:03.173112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.173825 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.173828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:03.174821 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:03.174832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.175305 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.175326 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:03.175415 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:03.175444 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:47:03.190914 (+ 15470us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 04:47:03.191137 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33145: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":101,"threads_started":1}
I20260924 04:47:03.191867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 5617
2026-09-24T04:47:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 04:47:03.229844 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:47:03.230732 392 webserver.cc:460] Webserver started at https://127.0.0.1:41415/ using document root <none> and password file <none>
I20260924 04:47:03.230834 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:47:03.231122 392 webserver.cc:460] Webserver started at http://127.0.0.1:44093/ using document root <none> and password file <none>
2026-09-24T04:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:03Z Disabled control of system clock
I20260924 04:47:03.239391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40845
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40845
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41415/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 04:47:03.344621 5720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:03.344872 5720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:03.344913 5720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:03.348940 5720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:03.348999 5720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:03.349014 5720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:03.349030 5720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:03.349043 5720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:47:03.349058 5720 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:47:03.349074 5720 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41415/jwks.json
I20260924 04:47:03.353818 5720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41415/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:03.354888 5720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:03.356259 5720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:03.364490 5725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:03.364470 5726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:03.364473 5728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:03.364737 5720 server_base.cc:1034] running on GCE node
I20260924 04:47:03.365274 5720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:03.366354 5720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:03.367530 5720 hybrid_clock.cc:648] HybridClock initialized: now 1790225223367490 us; error 38 us; skew 500 ppm
I20260924 04:47:03.369287 5720 webserver.cc:460] Webserver started at http://127.0.98.62:40119/ using document root <none> and password file <none>
I20260924 04:47:03.369820 5720 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:03.369884 5720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:03.370077 5720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:03.372061 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "4bdf44309ef04db49926e6995ac2d897"
format_stamp: "Formatted at 2026-09-24 04:47:03 on dist-test-slave-hvcl"
server_key: "a326edc06e2daa5bcaebc1c33ebf724a"
server_key_iv: "c77881fe063ba0aee324c6005eb79aea"
server_key_version: "encryptionkey@0"
I20260924 04:47:03.372589 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "4bdf44309ef04db49926e6995ac2d897"
format_stamp: "Formatted at 2026-09-24 04:47:03 on dist-test-slave-hvcl"
server_key: "a326edc06e2daa5bcaebc1c33ebf724a"
server_key_iv: "c77881fe063ba0aee324c6005eb79aea"
server_key_version: "encryptionkey@0"
I20260924 04:47:03.376137 5720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:47:03.378443 5734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:03.379644 5720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:03.379747 5720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "4bdf44309ef04db49926e6995ac2d897"
format_stamp: "Formatted at 2026-09-24 04:47:03 on dist-test-slave-hvcl"
server_key: "a326edc06e2daa5bcaebc1c33ebf724a"
server_key_iv: "c77881fe063ba0aee324c6005eb79aea"
server_key_version: "encryptionkey@0"
I20260924 04:47:03.379837 5720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:03.398550 5720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:03.399250 5720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:03.399420 5720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:03.405808 5720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40845
I20260924 04:47:03.405865 5786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40845 every 8 connection(s)
I20260924 04:47:03.406738 5720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:47:03.409869 5787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:03.414691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 5720
I20260924 04:47:03.414814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:47:03.415180 392 external_mini_cluster.cc:1428] Setting key 890cc7ea44078071e0c1ebe914955860
I20260924 04:47:03.415601 5787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897: Bootstrap starting.
I20260924 04:47:03.418052 5787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:03.418987 5787 log.cc:824] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:03.421473 5787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897: No bootstrap required, opened a new log
I20260924 04:47:03.425464 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:03.416536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.0.98.62:40845)
0924 04:47:03.417236 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:03.417251 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:03.417288 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:47:03.417925 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:03.417943 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:03.418440 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:03.418820 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:03.419332 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.420854 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.422221 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.422711 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.422852 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:03.423681 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:03.423686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:03.423699 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:03.424090 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:03.424146 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:03.424165 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:03.424239 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:03.424387 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 04:47:03.424646 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":135,"threads_started":1}
I20260924 04:47:03.425442 5787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bdf44309ef04db49926e6995ac2d897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40845 } }
I20260924 04:47:03.425704 5787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:03.425758 5787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bdf44309ef04db49926e6995ac2d897, State: Initialized, Role: FOLLOWER
I20260924 04:47:03.426164 5787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bdf44309ef04db49926e6995ac2d897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40845 } }
I20260924 04:47:03.426293 5787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:47:03.426352 5787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:47:03.426437 5787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:03.427541 5787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bdf44309ef04db49926e6995ac2d897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40845 } }
I20260924 04:47:03.427892 5787 leader_election.cc:302] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bdf44309ef04db49926e6995ac2d897; no voters:
I20260924 04:47:03.428226 5787 leader_election.cc:288] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:47:03.428380 5792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:03.428493 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:47:03.428637 5792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [term 1 LEADER]: Becoming Leader. State: Replica: 4bdf44309ef04db49926e6995ac2d897, State: Running, Role: LEADER
I20260924 04:47:03.428923 5792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bdf44309ef04db49926e6995ac2d897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40845 } }
I20260924 04:47:03.429628 5787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:03.430002 5792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bdf44309ef04db49926e6995ac2d897. Latest consensus state: current_term: 1 leader_uuid: "4bdf44309ef04db49926e6995ac2d897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bdf44309ef04db49926e6995ac2d897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40845 } } }
I20260924 04:47:03.430176 5792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:03.430485 5793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bdf44309ef04db49926e6995ac2d897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bdf44309ef04db49926e6995ac2d897" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40845 } } }
I20260924 04:47:03.430585 5793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:03.431216 5796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:03.433952 5796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:03.439181 5796 catalog_manager.cc:1347] Generated new cluster ID: b0b84082d33d4813bfd9fec0dcbff973
I20260924 04:47:03.439283 5796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:03.460716 5796 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:47:03.461614 5796 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:03.465302 5796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bdf44309ef04db49926e6995ac2d897: Generated new TSK 0
I20260924 04:47:03.465952 5796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:47:03.477310 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:03.472810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.0.98.62:40845)
0924 04:47:03.472919 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:03.472923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:03.473143 (+ 220us) server_negotiation.cc:407] Connection header received
0924 04:47:03.473254 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:03.473256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:03.473301 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:03.473386 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:47:03.473621 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.474160 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.475027 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.475255 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.475340 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:03.475624 (+ 284us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:47:03.476917 (+ 1293us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41415/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 04:47:03.477160 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42192: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41415/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 04:47:03.477380 5790 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42192: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41415/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 04:47:03.479312 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:03.472739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40845 (local address 127.0.0.1:42192)
0924 04:47:03.473007 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:03.473019 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:03.473126 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:03.473396 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:03.473400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:03.473408 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:47:03.473517 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:03.473523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.474265 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.474269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:03.474902 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:03.474910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.475341 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.475364 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:03.475451 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:03.475472 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:47:03.477078 (+ 1606us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41415/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 04:47:03.479137 (+ 2059us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40845: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41415/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":1760,"thread_start_us":101,"threads_started":1}
I20260924 04:47:03.479673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 5720
E20260924 04:47:03.488016 5817 webserver.cc:533] Webserver: sslize failed:
E20260924 04:47:03.488093 5817 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T04:47:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (287 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T04:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:03Z Disabled control of system clock
I20260924 04:47:03.525661 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45613
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45613
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/testchainca.cert with env {}
W20260924 04:47:03.633673 5821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:03.633950 5821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:03.634004 5821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:03.638036 5821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:03.638104 5821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:03.638125 5821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:03.638145 5821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:03.638165 5821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:47:03.638197 5821 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:47:03.638223 5821 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 04:47:03.643587 5821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:03.644591 5821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:03.645552 5821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:03.653590 5829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:03.653610 5827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:03.653610 5826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:03.654080 5821 server_base.cc:1034] running on GCE node
I20260924 04:47:03.654587 5821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:03.655719 5821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:03.656883 5821 hybrid_clock.cc:648] HybridClock initialized: now 1790225223656868 us; error 26 us; skew 500 ppm
I20260924 04:47:03.658726 5821 webserver.cc:460] Webserver started at http://127.0.98.62:33881/ using document root <none> and password file <none>
I20260924 04:47:03.659322 5821 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:03.659382 5821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:03.659556 5821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:03.661572 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "b3542b00d558407eb9bfddac71c9b937"
format_stamp: "Formatted at 2026-09-24 04:47:03 on dist-test-slave-hvcl"
server_key: "ae30fc95521efc8223b5e5ac43180eaf"
server_key_iv: "080800c90b6f050935fdfb2a8939c8bc"
server_key_version: "encryptionkey@0"
I20260924 04:47:03.662099 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "b3542b00d558407eb9bfddac71c9b937"
format_stamp: "Formatted at 2026-09-24 04:47:03 on dist-test-slave-hvcl"
server_key: "ae30fc95521efc8223b5e5ac43180eaf"
server_key_iv: "080800c90b6f050935fdfb2a8939c8bc"
server_key_version: "encryptionkey@0"
I20260924 04:47:03.665824 5821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 04:47:03.668277 5835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:03.669517 5821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:47:03.669628 5821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "b3542b00d558407eb9bfddac71c9b937"
format_stamp: "Formatted at 2026-09-24 04:47:03 on dist-test-slave-hvcl"
server_key: "ae30fc95521efc8223b5e5ac43180eaf"
server_key_iv: "080800c90b6f050935fdfb2a8939c8bc"
server_key_version: "encryptionkey@0"
I20260924 04:47:03.669728 5821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:03.686327 5821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:03.687065 5821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:03.687245 5821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:03.693743 5821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45613
I20260924 04:47:03.693756 5887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45613 every 8 connection(s)
I20260924 04:47:03.694626 5821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:47:03.697649 5888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:03.700927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 5821
I20260924 04:47:03.701035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:47:03.701280 392 external_mini_cluster.cc:1428] Setting key 841ad6bf7834d6a8099fcf8669322485
I20260924 04:47:03.703290 5888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937: Bootstrap starting.
I20260924 04:47:03.705947 5888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:03.706813 5888 log.cc:824] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:03.708999 5888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937: No bootstrap required, opened a new log
I20260924 04:47:03.712100 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:03.702572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44122 (local address 127.0.98.62:45613)
0924 04:47:03.703259 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:03.703280 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:03.703322 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:47:03.704128 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:03.704196 (+ 68us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:03.704709 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:03.705073 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:03.705510 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.706953 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.708074 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.708783 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.708939 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:03.709737 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:03.709744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:03.709759 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:03.710242 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:03.710319 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:03.710345 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:03.710434 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:03.710582 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 04:47:03.710857 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
I20260924 04:47:03.712790 5888 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3542b00d558407eb9bfddac71c9b937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45613 } }
I20260924 04:47:03.712996 5888 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:03.713075 5888 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3542b00d558407eb9bfddac71c9b937, State: Initialized, Role: FOLLOWER
I20260924 04:47:03.713527 5888 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3542b00d558407eb9bfddac71c9b937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45613 } }
I20260924 04:47:03.713683 5888 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:47:03.713744 5888 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:47:03.713836 5888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:03.714864 5888 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3542b00d558407eb9bfddac71c9b937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45613 } }
I20260924 04:47:03.714957 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:47:03.715211 5888 leader_election.cc:302] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3542b00d558407eb9bfddac71c9b937; no voters:
I20260924 04:47:03.715489 5888 leader_election.cc:288] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:47:03.715620 5893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:03.715862 5893 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [term 1 LEADER]: Becoming Leader. State: Replica: b3542b00d558407eb9bfddac71c9b937, State: Running, Role: LEADER
I20260924 04:47:03.716210 5893 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3542b00d558407eb9bfddac71c9b937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45613 } }
I20260924 04:47:03.716701 5888 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:03.717746 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3542b00d558407eb9bfddac71c9b937. Latest consensus state: current_term: 1 leader_uuid: "b3542b00d558407eb9bfddac71c9b937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3542b00d558407eb9bfddac71c9b937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45613 } } }
I20260924 04:47:03.717903 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:03.718178 5894 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3542b00d558407eb9bfddac71c9b937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3542b00d558407eb9bfddac71c9b937" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45613 } } }
I20260924 04:47:03.718255 5902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:03.718335 5894 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:03.721722 5902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:03.727089 5902 catalog_manager.cc:1347] Generated new cluster ID: 482674cf84744c1b9449ff6c14adfcf8
I20260924 04:47:03.727155 5902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:03.741681 5902 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:47:03.742995 5902 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:03.759804 5902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3542b00d558407eb9bfddac71c9b937: Generated new TSK 0
I20260924 04:47:03.760685 5902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:47:03.847838 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:03.773501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.0.98.62:45613)
0924 04:47:03.773639 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:03.773644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:03.773856 (+ 212us) server_negotiation.cc:407] Connection header received
0924 04:47:03.774005 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:03.774007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:03.774064 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:03.774153 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:47:03.774405 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.774956 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.776213 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.776451 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.776542 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:03.776901 (+ 359us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:47:03.847430 (+ 70529us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 04:47:03.847701 (+ 271us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44128: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 04:47:03.847918 5891 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44128: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 04:47:03.848119 5917 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:03.773441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45613 (local address 127.0.0.1:44128)
0924 04:47:03.773765 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:03.773780 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:03.773897 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:03.774167 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:03.774170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:03.774179 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:47:03.774296 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:03.774303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.775085 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.775088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:03.776074 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:03.776084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:03.776549 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:03.776573 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:03.776675 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:03.776704 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:47:03.847705 (+ 71001us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 04:47:03.847919 (+ 214us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45613: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":137,"threads_started":1}
I20260924 04:47:03.848744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 5821
2026-09-24T04:47:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (348 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:47:04 dist-test-slave-hvcl krb5kdc[5929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:47:04 dist-test-slave-hvcl krb5kdc[5929](info): set up 2 sockets
Sep 24 04:47:04 dist-test-slave-hvcl krb5kdc[5929](info): commencing operation
krb5kdc: starting...
W20260924 04:47:05.055029 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:47:05 dist-test-slave-hvcl krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:47:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:47:05.172107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37405
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:05.279613 5942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:05.279901 5942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:05.279965 5942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:05.284080 5942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:05.284148 5942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:05.284174 5942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:05.284196 5942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:05.284216 5942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:47:05.289726 5942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:05.290843 5942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:05.291936 5942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:05.300336 5948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:05.300348 5950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:05.300583 5942 server_base.cc:1034] running on GCE node
W20260924 04:47:05.300412 5947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:05.301257 5942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:05.302416 5942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:05.303591 5942 hybrid_clock.cc:648] HybridClock initialized: now 1790225225303561 us; error 44 us; skew 500 ppm
Sep 24 04:47:05 dist-test-slave-hvcl krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:05.306463 5942 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:47:05.307441 5942 webserver.cc:460] Webserver started at http://127.0.98.62:34689/ using document root <none> and password file <none>
I20260924 04:47:05.308005 5942 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:05.308070 5942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:05.308270 5942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:05.310302 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "bd13cbc116544995a3dd82a805f6168d"
format_stamp: "Formatted at 2026-09-24 04:47:05 on dist-test-slave-hvcl"
server_key: "b8454c457aa1a5e97c823ab5bb117a63"
server_key_iv: "9ab7c7e9039ee8ca531f887db8da722c"
server_key_version: "encryptionkey@0"
I20260924 04:47:05.310854 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd13cbc116544995a3dd82a805f6168d"
format_stamp: "Formatted at 2026-09-24 04:47:05 on dist-test-slave-hvcl"
server_key: "b8454c457aa1a5e97c823ab5bb117a63"
server_key_iv: "9ab7c7e9039ee8ca531f887db8da722c"
server_key_version: "encryptionkey@0"
I20260924 04:47:05.314536 5942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:47:05.317026 5957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:05.318333 5942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:05.318455 5942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "bd13cbc116544995a3dd82a805f6168d"
format_stamp: "Formatted at 2026-09-24 04:47:05 on dist-test-slave-hvcl"
server_key: "b8454c457aa1a5e97c823ab5bb117a63"
server_key_iv: "9ab7c7e9039ee8ca531f887db8da722c"
server_key_version: "encryptionkey@0"
I20260924 04:47:05.318562 5942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:05.336675 5942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:05.339864 5942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:05.340054 5942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:05.346447 5942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37405
I20260924 04:47:05.346724 6009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37405 every 8 connection(s)
I20260924 04:47:05.347424 5942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:47:05.348161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 5942
I20260924 04:47:05.348278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:47:05.348536 392 external_mini_cluster.cc:1428] Setting key 926f666f508b8fc356a8109f913b5049
I20260924 04:47:05.350598 6010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:05.357101 6010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d: Bootstrap starting.
Sep 24 04:47:05 dist-test-slave-hvcl krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:47:05.359582 6010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:05.360500 6010 log.cc:824] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:05.362872 6010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d: No bootstrap required, opened a new log
I20260924 04:47:05.363655 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:05.349866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.0.98.62:37405)
0924 04:47:05.350290 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:05.350310 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:05.350352 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:47:05.351206 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:05.351229 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:05.351646 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:05.352022 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:05.352698 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:05.354182 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:05.355115 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:05.355800 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:05.355956 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:05.358344 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:05.358350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:05.358366 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:05.358398 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:05.360395 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:05.360925 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:05.360931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:05.360938 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:05.361048 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:05.361356 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:05.361360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:05.361361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:05.361878 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:05.362056 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:05.362335 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 04:47:05.362629 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20260924 04:47:05.366866 6010 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd13cbc116544995a3dd82a805f6168d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37405 } }
I20260924 04:47:05.367151 6010 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:05.367201 6010 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd13cbc116544995a3dd82a805f6168d, State: Initialized, Role: FOLLOWER
I20260924 04:47:05.367591 6010 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd13cbc116544995a3dd82a805f6168d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37405 } }
I20260924 04:47:05.367704 6010 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:47:05.367753 6010 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:47:05.367825 6010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:05.368860 6010 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd13cbc116544995a3dd82a805f6168d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37405 } }
I20260924 04:47:05.369217 6010 leader_election.cc:302] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd13cbc116544995a3dd82a805f6168d; no voters:
I20260924 04:47:05.369503 6010 leader_election.cc:288] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:47:05.369627 6015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:05.369829 6015 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [term 1 LEADER]: Becoming Leader. State: Replica: bd13cbc116544995a3dd82a805f6168d, State: Running, Role: LEADER
I20260924 04:47:05.370217 6015 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd13cbc116544995a3dd82a805f6168d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37405 } }
I20260924 04:47:05.370627 6010 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:05.371853 6017 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd13cbc116544995a3dd82a805f6168d. Latest consensus state: current_term: 1 leader_uuid: "bd13cbc116544995a3dd82a805f6168d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd13cbc116544995a3dd82a805f6168d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37405 } } }
I20260924 04:47:05.371984 6017 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:05.371874 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd13cbc116544995a3dd82a805f6168d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd13cbc116544995a3dd82a805f6168d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37405 } } }
I20260924 04:47:05.372118 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:05.372401 6025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:05.375682 6025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:05.381186 6025 catalog_manager.cc:1347] Generated new cluster ID: 92c9aae961a740e9acadae9edd6d98f4
I20260924 04:47:05.381254 6025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:05.395808 6025 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:47:05.397089 6025 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:05.420336 6025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd13cbc116544995a3dd82a805f6168d: Generated new TSK 0
I20260924 04:47:05.421305 6025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:47:05.458230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37405
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:05.570041 6036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:05.570389 6036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:05.570442 6036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:05.574576 6036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:05.574628 6036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:05.574688 6036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:47:05.579830 6036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:05.580825 6036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:05.581789 6036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:05.591202 6041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:05.591224 6036 server_base.cc:1034] running on GCE node
W20260924 04:47:05.591459 6042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:05.591204 6044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:05.592427 6036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:05.593097 6036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:05.594259 6036 hybrid_clock.cc:648] HybridClock initialized: now 1790225225594239 us; error 33 us; skew 500 ppm
Sep 24 04:47:05 dist-test-slave-hvcl krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:05.601495 6036 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:47:05.602478 6036 webserver.cc:460] Webserver started at http://127.0.98.1:45581/ using document root <none> and password file <none>
I20260924 04:47:05.603080 6036 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:05.603152 6036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:05.603353 6036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:05.605556 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "28618142a7ac449fa753a1bce46fa256"
format_stamp: "Formatted at 2026-09-24 04:47:05 on dist-test-slave-hvcl"
server_key: "0513ef8c7c1664db1ff9db9f17b69d99"
server_key_iv: "d21e56a6f24f7054578edcc1abce5a82"
server_key_version: "encryptionkey@0"
I20260924 04:47:05.606143 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "28618142a7ac449fa753a1bce46fa256"
format_stamp: "Formatted at 2026-09-24 04:47:05 on dist-test-slave-hvcl"
server_key: "0513ef8c7c1664db1ff9db9f17b69d99"
server_key_iv: "d21e56a6f24f7054578edcc1abce5a82"
server_key_version: "encryptionkey@0"
I20260924 04:47:05.609974 6036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 04:47:05.612409 6051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:05.613654 6036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:47:05.613766 6036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "28618142a7ac449fa753a1bce46fa256"
format_stamp: "Formatted at 2026-09-24 04:47:05 on dist-test-slave-hvcl"
server_key: "0513ef8c7c1664db1ff9db9f17b69d99"
server_key_iv: "d21e56a6f24f7054578edcc1abce5a82"
server_key_version: "encryptionkey@0"
I20260924 04:47:05.613862 6036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:05.647329 6036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:05.650554 6036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:05.650731 6036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:05.651319 6036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:05.652137 6036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:05.652184 6036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:05.652235 6036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:05.652264 6036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:05.660568 6036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43007
I20260924 04:47:05.660599 6164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43007 every 8 connection(s)
I20260924 04:47:05.661437 6036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:47:05.665102 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 6036
I20260924 04:47:05.665215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:47:05.665493 392 external_mini_cluster.cc:1428] Setting key 2f39c5a6563c4ef135d3f1b53d9cb7b3
Sep 24 04:47:05 dist-test-slave-hvcl krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:47:05.675496 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:05.663323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50397 (local address 127.0.98.62:37405)
0924 04:47:05.663495 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:05.663499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:05.664394 (+ 895us) server_negotiation.cc:407] Connection header received
0924 04:47:05.665778 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:05.665781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:05.665835 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:05.665909 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:05.667098 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:05.667664 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:05.668766 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:05.669047 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:05.669134 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:05.671875 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:05.671879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:05.671882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:05.671909 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:05.673357 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:05.673923 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:05.673927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:05.673929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:05.674011 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:05.674325 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:05.674329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:05.674331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:05.674549 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:05.674644 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:05.675251 (+ 607us) server_negotiation.cc:299] Negotiation successful
0924 04:47:05.675358 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:47:05.676440 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:05.663736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37405 (local address 127.0.98.1:50397)
0924 04:47:05.664199 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:05.664242 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:05.665567 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:05.666121 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:05.666128 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:05.666467 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:05.666938 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:05.666952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:05.667779 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:05.667782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:05.668650 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:05.668659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:05.669181 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:05.669186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:05.669343 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:05.669900 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:05.669915 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:05.671671 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:05.673470 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:05.673476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:05.673493 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:05.673810 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:05.674098 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:05.674101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:05.674104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:05.674237 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:05.674679 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:05.674686 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:05.674946 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 04:47:05.675190 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 04:47:05.675455 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":110,"threads_started":1}
I20260924 04:47:05.677678 6165 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37405
I20260924 04:47:05.677901 6165 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:05.678377 6165 heartbeater.cc:511] Master 127.0.98.62:37405 requested a full tablet report, sending...
I20260924 04:47:05.679867 5973 ts_manager.cc:194] Registered new tserver with Master: 28618142a7ac449fa753a1bce46fa256 (127.0.98.1:43007)
I20260924 04:47:05.680984 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:47:05.692669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37405
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:05.800401 6170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:05.800678 6170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:05.800720 6170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:05.804833 6170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:05.804911 6170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:05.804981 6170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:47:05.810527 6170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:05.811653 6170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:05.812660 6170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:47:05.822127 6170 server_base.cc:1034] running on GCE node
W20260924 04:47:05.822206 6176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:05.822091 6178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:05.822104 6175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:05.822708 6170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:05.823386 6170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:05.824555 6170 hybrid_clock.cc:648] HybridClock initialized: now 1790225225824525 us; error 41 us; skew 500 ppm
Sep 24 04:47:05 dist-test-slave-hvcl krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:05.827270 6170 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:47:05.828120 6170 webserver.cc:460] Webserver started at http://127.0.98.2:46459/ using document root <none> and password file <none>
I20260924 04:47:05.828632 6170 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:05.828687 6170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:05.828904 6170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:05.830865 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "827b26fbb0864b4b8c3ff73ed08c16ea"
format_stamp: "Formatted at 2026-09-24 04:47:05 on dist-test-slave-hvcl"
server_key: "e25219870c717457bb828d2cb2a4bc70"
server_key_iv: "16d0fe15e4b7956db2021c2756a132bd"
server_key_version: "encryptionkey@0"
I20260924 04:47:05.831414 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "827b26fbb0864b4b8c3ff73ed08c16ea"
format_stamp: "Formatted at 2026-09-24 04:47:05 on dist-test-slave-hvcl"
server_key: "e25219870c717457bb828d2cb2a4bc70"
server_key_iv: "16d0fe15e4b7956db2021c2756a132bd"
server_key_version: "encryptionkey@0"
I20260924 04:47:05.834954 6170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:47:05.837399 6185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:05.838593 6170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:47:05.838711 6170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "827b26fbb0864b4b8c3ff73ed08c16ea"
format_stamp: "Formatted at 2026-09-24 04:47:05 on dist-test-slave-hvcl"
server_key: "e25219870c717457bb828d2cb2a4bc70"
server_key_iv: "16d0fe15e4b7956db2021c2756a132bd"
server_key_version: "encryptionkey@0"
I20260924 04:47:05.838805 6170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:05.852523 6170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:05.855654 6170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:05.855826 6170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:05.856398 6170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:05.857189 6170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:05.857235 6170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:05.857286 6170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:05.857316 6170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:05.865932 6170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43461
I20260924 04:47:05.865975 6298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43461 every 8 connection(s)
I20260924 04:47:05.866822 6170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:47:05.868875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 6170
I20260924 04:47:05.868979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:47:05.869290 392 external_mini_cluster.cc:1428] Setting key c87833ad265b5e7d91a8a706988e965a
Sep 24 04:47:05 dist-test-slave-hvcl krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:47:05.881292 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:05.868757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53781 (local address 127.0.98.62:37405)
0924 04:47:05.868921 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:05.868926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:05.869872 (+ 946us) server_negotiation.cc:407] Connection header received
0924 04:47:05.871452 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:05.871455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:05.871509 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:05.871580 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:05.872939 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:05.873547 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:05.874739 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:05.875001 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:05.875091 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:05.877796 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:05.877799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:05.877801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:05.877826 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:05.879218 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:05.879777 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:05.879780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:05.879781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:05.879857 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:05.880132 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:05.880135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:05.880137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:05.880344 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:05.880416 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:05.881025 (+ 609us) server_negotiation.cc:299] Negotiation successful
0924 04:47:05.881130 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:47:05.882167 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:05.869188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37405 (local address 127.0.98.2:53781)
0924 04:47:05.869722 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:05.869775 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:05.871230 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:05.871773 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:05.871783 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:05.872204 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:05.872778 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:05.872798 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:05.873682 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:05.873685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:05.874604 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:05.874613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:05.875142 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:05.875146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:05.875305 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:05.875863 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:05.875879 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:05.877597 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:05.879348 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:05.879353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:05.879367 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:05.879684 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:05.879927 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:05.879929 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:05.879930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:05.880066 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:05.880434 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:05.880439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:05.880737 (+ 298us) client_negotiation.cc:769] Sending connection context
0924 04:47:05.880984 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 04:47:05.881302 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":142,"threads_started":1}
I20260924 04:47:05.883335 6299 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37405
I20260924 04:47:05.883601 6299 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:05.884166 6299 heartbeater.cc:511] Master 127.0.98.62:37405 requested a full tablet report, sending...
I20260924 04:47:05.885221 5973 ts_manager.cc:194] Registered new tserver with Master: 827b26fbb0864b4b8c3ff73ed08c16ea (127.0.98.2:43461)
I20260924 04:47:05.885785 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:47:05.898221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37405
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:06.007869 6304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:06.008129 6304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:06.008172 6304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:06.012146 6304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:06.012202 6304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:06.012261 6304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:47:06.017485 6304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:06.018539 6304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:06.019503 6304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:06.028834 6310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:06.028841 6309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:06.028829 6312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:06.028925 6304 server_base.cc:1034] running on GCE node
I20260924 04:47:06.029505 6304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:06.030120 6304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:06.031271 6304 hybrid_clock.cc:648] HybridClock initialized: now 1790225226031257 us; error 30 us; skew 500 ppm
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:06.033939 6304 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:47:06.034796 6304 webserver.cc:460] Webserver started at http://127.0.98.3:42255/ using document root <none> and password file <none>
I20260924 04:47:06.035352 6304 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:06.035414 6304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:06.035604 6304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:06.037640 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "88c16365e82c4315a5e6cc3da6f76f7a"
format_stamp: "Formatted at 2026-09-24 04:47:06 on dist-test-slave-hvcl"
server_key: "6b5e7c78f9434baed6ecac364e2cd24a"
server_key_iv: "9a1e999776c45999a3924944a6bea982"
server_key_version: "encryptionkey@0"
I20260924 04:47:06.038172 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "88c16365e82c4315a5e6cc3da6f76f7a"
format_stamp: "Formatted at 2026-09-24 04:47:06 on dist-test-slave-hvcl"
server_key: "6b5e7c78f9434baed6ecac364e2cd24a"
server_key_iv: "9a1e999776c45999a3924944a6bea982"
server_key_version: "encryptionkey@0"
I20260924 04:47:06.041945 6304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 04:47:06.044435 6319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:06.045625 6304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:47:06.045732 6304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "88c16365e82c4315a5e6cc3da6f76f7a"
format_stamp: "Formatted at 2026-09-24 04:47:06 on dist-test-slave-hvcl"
server_key: "6b5e7c78f9434baed6ecac364e2cd24a"
server_key_iv: "9a1e999776c45999a3924944a6bea982"
server_key_version: "encryptionkey@0"
I20260924 04:47:06.045831 6304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:06.058310 6304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:06.061659 6304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:06.061838 6304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:06.062445 6304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:06.063297 6304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:06.063344 6304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:06.063386 6304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:06.063414 6304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:06.072247 6304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34265
I20260924 04:47:06.072300 6432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34265 every 8 connection(s)
I20260924 04:47:06.073153 6304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:47:06.074425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 6304
I20260924 04:47:06.074546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:47:06.074858 392 external_mini_cluster.cc:1428] Setting key 41745652d3696184fcc6861c6406f860
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:47:06.086416 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:06.075078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56749 (local address 127.0.98.62:37405)
0924 04:47:06.075212 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:06.075217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:06.076236 (+ 1019us) server_negotiation.cc:407] Connection header received
0924 04:47:06.077656 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:06.077659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:06.077713 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:06.077786 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:06.079022 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:06.079546 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:06.080662 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:06.080874 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:06.080956 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:06.083306 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:06.083309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:06.083311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:06.083335 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:06.084699 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:06.085164 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:06.085167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:06.085168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:06.085253 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:06.085503 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:06.085505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:06.085506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:06.085709 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:06.085778 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:06.086202 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 04:47:06.086294 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:47:06.087379 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:06.075567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37405 (local address 127.0.98.3:56749)
0924 04:47:06.076097 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:06.076154 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:06.077438 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:06.077974 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:06.077982 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:06.078360 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:06.078869 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:06.078884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:06.079660 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:06.079663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:06.080549 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:06.080557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:06.081001 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:06.081005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:06.081169 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:06.081714 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:06.081730 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:06.083148 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:06.084804 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:06.084809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:06.084822 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:06.085074 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:06.085321 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:06.085323 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:06.085324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:06.085444 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:06.085779 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:06.085783 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:06.086013 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 04:47:06.086180 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 04:47:06.086424 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":132,"threads_started":1}
I20260924 04:47:06.088375 6433 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37405
I20260924 04:47:06.088641 6433 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:06.089170 6433 heartbeater.cc:511] Master 127.0.98.62:37405 requested a full tablet report, sending...
I20260924 04:47:06.090183 5974 ts_manager.cc:194] Registered new tserver with Master: 88c16365e82c4315a5e6cc3da6f76f7a (127.0.98.3:34265)
I20260924 04:47:06.090829 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56749
I20260924 04:47:06.100703 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/test-admin.keytab.
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:06.115927 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 04:47:06.118614 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:06.117193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.0.98.1:43007)
0924 04:47:06.117466 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:06.117480 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:06.117505 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:47:06.117558 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:06.117564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:06.117761 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:06.117883 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:06.118454 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35146: BlockingRecv error: recv got EOF from 127.0.0.1:35146 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
W20260924 04:47:06.118935 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/krb5cc)
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:06.119913 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:06.120780 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:47:06.130182 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:06.121678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.0.98.1:43007)
0924 04:47:06.121798 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:06.121802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:06.121866 (+ 64us) server_negotiation.cc:407] Connection header received
0924 04:47:06.121993 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:06.122022 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:06.122064 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:06.122131 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:06.122736 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:06.123596 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:06.124407 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:06.124786 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:06.124880 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:06.127067 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:06.127077 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:06.127089 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:06.127114 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:06.128629 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:06.128984 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:06.128988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:06.128989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:06.129087 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:06.129295 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:06.129297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:06.129298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:06.129527 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:06.129678 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:06.129901 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 04:47:06.130042 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 04:47:06.133904 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:06.133010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35162 (local address 127.0.98.1:43007)
0924 04:47:06.133127 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:06.133131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:06.133199 (+ 68us) server_negotiation.cc:407] Connection header received
0924 04:47:06.133307 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:06.133310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:06.133347 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:06.133406 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:06.133807 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35162: BlockingRecv error: recv got EOF from 127.0.0.1:35162 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 04:47:06.134042 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/krb5cc)
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:06.134949 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:06.135923 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:47:06.144155 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:06.136730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.0.98.1:43007)
0924 04:47:06.136864 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:06.136866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:06.136919 (+ 53us) server_negotiation.cc:407] Connection header received
0924 04:47:06.137006 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:06.137008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:06.137040 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:06.137102 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:06.137653 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:06.138167 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:06.138950 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:06.139227 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:06.139309 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:06.141382 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:06.141385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:06.141387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:06.141407 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:06.142811 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:06.143222 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:06.143224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:06.143226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:06.143294 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:06.143471 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:06.143473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:06.143474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:06.143662 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:06.143754 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:06.143916 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 04:47:06.144001 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 04:47:06.146541 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:06.145764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.0.98.1:43007)
0924 04:47:06.145886 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:06.145889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:06.145951 (+ 62us) server_negotiation.cc:407] Connection header received
0924 04:47:06.146029 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:06.146031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:06.146067 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:06.146124 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:06.146450 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35180: BlockingRecv error: recv got EOF from 127.0.0.1:35180 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 04:47:06.146652 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225194677594-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:06.147635 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:06.148406 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:47:06.156247 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:06.149242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35186 (local address 127.0.98.1:43007)
0924 04:47:06.149342 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:06.149347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:06.149461 (+ 114us) server_negotiation.cc:407] Connection header received
0924 04:47:06.149545 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:06.149548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:06.149585 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:06.149645 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:06.150129 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:06.150612 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:06.151402 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:06.151607 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:06.151688 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:06.153770 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:06.153773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:06.153775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:06.153795 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:06.154870 (+ 1075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:06.155244 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:06.155247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:06.155249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:06.155315 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:06.155544 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:06.155547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:06.155549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:06.155693 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:06.155822 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:06.156025 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 04:47:06.156119 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:47:06.157152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 6036
I20260924 04:47:06.161846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 6170
I20260924 04:47:06.166242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 6304
I20260924 04:47:06.172597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 5942
2026-09-24T04:47:06Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2081 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[6453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[6453](info): set up 2 sockets
Sep 24 04:47:06 dist-test-slave-hvcl krb5kdc[6453](info): commencing operation
krb5kdc: starting...
W20260924 04:47:07.135087 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:47:07 dist-test-slave-hvcl krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:47:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 04:47:07.254539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42905
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:07.362018 6466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:07.362288 6466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:07.362341 6466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:07.366374 6466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:07.366442 6466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:07.366469 6466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:07.366489 6466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:07.366511 6466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:47:07.371754 6466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:07.372815 6466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:07.373811 6466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:07.381812 6474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:07.381852 6472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:07.382061 6471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:07.382266 6466 server_base.cc:1034] running on GCE node
I20260924 04:47:07.382807 6466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:07.383880 6466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:07.385048 6466 hybrid_clock.cc:648] HybridClock initialized: now 1790225227385023 us; error 39 us; skew 500 ppm
Sep 24 04:47:07 dist-test-slave-hvcl krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:07.387710 6466 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 04:47:07.388535 6466 webserver.cc:460] Webserver started at http://127.0.98.62:37187/ using document root <none> and password file <none>
I20260924 04:47:07.389087 6466 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:07.389142 6466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:07.389302 6466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:07.391337 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "362ebf6fc3494b56afca08f7ea649efb"
format_stamp: "Formatted at 2026-09-24 04:47:07 on dist-test-slave-hvcl"
server_key: "87b03abd53cd73753a23aac00f088349"
server_key_iv: "6427a644fc7f7ca2f5557e426b11be09"
server_key_version: "encryptionkey@0"
I20260924 04:47:07.391858 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "362ebf6fc3494b56afca08f7ea649efb"
format_stamp: "Formatted at 2026-09-24 04:47:07 on dist-test-slave-hvcl"
server_key: "87b03abd53cd73753a23aac00f088349"
server_key_iv: "6427a644fc7f7ca2f5557e426b11be09"
server_key_version: "encryptionkey@0"
I20260924 04:47:07.395669 6466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:47:07.398219 6481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:07.399554 6466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 04:47:07.399704 6466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "362ebf6fc3494b56afca08f7ea649efb"
format_stamp: "Formatted at 2026-09-24 04:47:07 on dist-test-slave-hvcl"
server_key: "87b03abd53cd73753a23aac00f088349"
server_key_iv: "6427a644fc7f7ca2f5557e426b11be09"
server_key_version: "encryptionkey@0"
I20260924 04:47:07.399803 6466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:07.419759 6466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:07.423352 6466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:07.423516 6466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:07.430491 6466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42905
I20260924 04:47:07.430794 6533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42905 every 8 connection(s)
I20260924 04:47:07.431548 6466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:47:07.435138 6534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:07.440744 6534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb: Bootstrap starting.
I20260924 04:47:07.440820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 6466
I20260924 04:47:07.440940 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:47:07.441202 392 external_mini_cluster.cc:1428] Setting key ad9a109779e7595f100980ea2522a963
I20260924 04:47:07.443356 6534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:07.444060 6534 log.cc:824] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:07.446107 6534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb: No bootstrap required, opened a new log
Sep 24 04:47:07 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:47:07.450268 6534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } }
I20260924 04:47:07.450536 6534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:07.450603 6534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362ebf6fc3494b56afca08f7ea649efb, State: Initialized, Role: FOLLOWER
I20260924 04:47:07.451103 6534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } }
I20260924 04:47:07.451272 6534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:47:07.451335 6534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:47:07.451427 6534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:07.452316 6534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } }
I20260924 04:47:07.452637 6534 leader_election.cc:302] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 362ebf6fc3494b56afca08f7ea649efb; no voters:
I20260924 04:47:07.452942 6534 leader_election.cc:288] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:47:07.453027 6539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:07.453222 6539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [term 1 LEADER]: Becoming Leader. State: Replica: 362ebf6fc3494b56afca08f7ea649efb, State: Running, Role: LEADER
I20260924 04:47:07.453670 6539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } }
I20260924 04:47:07.454270 6534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:07.454825 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 362ebf6fc3494b56afca08f7ea649efb. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } }
I20260924 04:47:07.454937 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:07.455333 6545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:07.455365 6540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } }
I20260924 04:47:07.455529 6540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:07.456178 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:07.442509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.0.98.62:42905)
0924 04:47:07.442929 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:07.442945 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:07.443010 (+ 65us) server_negotiation.cc:407] Connection header received
0924 04:47:07.443742 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:07.443762 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:07.444116 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:07.444442 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:07.445198 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:07.446263 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:07.447429 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:07.447973 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:07.448109 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:07.450883 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:07.450890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:07.450907 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:07.450938 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:07.452701 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:07.453156 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:07.453162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:07.453169 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:07.453320 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:07.453618 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:07.453621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:07.453623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:07.454056 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:07.454245 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:07.454815 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 04:47:07.455149 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
I20260924 04:47:07.459203 6545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:07.464105 6545 catalog_manager.cc:1347] Generated new cluster ID: 2723da5504bc4f1a893a21ee44f47e03
I20260924 04:47:07.464172 6545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:07.493858 6545 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:47:07.494843 6545 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:07.498425 6545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb: Generated new TSK 0
I20260924 04:47:07.499152 6545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 04:47:07.529868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42905
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:07.645288 6560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:07.645543 6560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:07.645586 6560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:07.649559 6560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:07.649614 6560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:07.649674 6560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:47:07.654992 6560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:07.656028 6560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:07.657027 6560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:07.666538 6566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:07.666517 6568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:07.666666 6560 server_base.cc:1034] running on GCE node
W20260924 04:47:07.666572 6565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:07.667142 6560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:07.667788 6560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:07.668951 6560 hybrid_clock.cc:648] HybridClock initialized: now 1790225227668933 us; error 32 us; skew 500 ppm
Sep 24 04:47:07 dist-test-slave-hvcl krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:07.671522 6560 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:47:07.672380 6560 webserver.cc:460] Webserver started at http://127.0.98.1:39941/ using document root <none> and password file <none>
I20260924 04:47:07.672967 6560 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:07.673032 6560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:07.673228 6560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:07.675422 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "3ca550e78f9e4458bfe8316441d9e1af"
format_stamp: "Formatted at 2026-09-24 04:47:07 on dist-test-slave-hvcl"
server_key: "99ddda11c3f7ae9f6613d2949e734820"
server_key_iv: "fb65d00dcfbde9a3e6f5631ba141604a"
server_key_version: "encryptionkey@0"
I20260924 04:47:07.675981 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3ca550e78f9e4458bfe8316441d9e1af"
format_stamp: "Formatted at 2026-09-24 04:47:07 on dist-test-slave-hvcl"
server_key: "99ddda11c3f7ae9f6613d2949e734820"
server_key_iv: "fb65d00dcfbde9a3e6f5631ba141604a"
server_key_version: "encryptionkey@0"
I20260924 04:47:07.679407 6560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:47:07.681645 6575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:07.683034 6560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:07.683143 6560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "3ca550e78f9e4458bfe8316441d9e1af"
format_stamp: "Formatted at 2026-09-24 04:47:07 on dist-test-slave-hvcl"
server_key: "99ddda11c3f7ae9f6613d2949e734820"
server_key_iv: "fb65d00dcfbde9a3e6f5631ba141604a"
server_key_version: "encryptionkey@0"
I20260924 04:47:07.683239 6560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:07.699720 6560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:07.703171 6560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:07.703347 6560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:07.703961 6560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:07.704787 6560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:07.704842 6560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:07.704896 6560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:07.704921 6560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:07.712971 6560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43471
I20260924 04:47:07.712996 6688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43471 every 8 connection(s)
I20260924 04:47:07.713821 6560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:47:07.714618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 6560
I20260924 04:47:07.714745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:47:07.715070 392 external_mini_cluster.cc:1428] Setting key b3f7f03be9dd84b54c39f8beb459620a
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 24 04:47:07 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260924 04:47:07.728643 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:07.715679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57015 (local address 127.0.98.62:42905)
0924 04:47:07.715834 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:07.715838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:07.716724 (+ 886us) server_negotiation.cc:407] Connection header received
0924 04:47:07.718122 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:07.718125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:07.718170 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:07.718257 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:07.719790 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:07.720322 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:07.721988 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:07.722265 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:07.722360 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:07.725101 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:07.725104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:07.725106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:07.725132 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:07.726380 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:07.726949 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:07.726953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:07.726954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:07.727115 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:07.727499 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:07.727502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:07.727503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:07.727702 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:07.727785 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:07.728376 (+ 591us) server_negotiation.cc:299] Negotiation successful
0924 04:47:07.728474 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 04:47:07.729605 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:07.716100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.1:57015)
0924 04:47:07.716596 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:07.716638 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:07.717873 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:07.718456 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:07.718465 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:07.718995 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:07.719626 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:07.719644 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:07.720442 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:07.720446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:07.721844 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:07.721857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:07.722425 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:07.722429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:07.722633 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:07.723309 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:07.723332 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:07.724938 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:07.726503 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:07.726507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:07.726520 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:07.726825 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:07.727210 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:07.727213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:07.727215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:07.727385 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:07.727808 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:07.727815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:07.728132 (+ 317us) client_negotiation.cc:769] Sending connection context
0924 04:47:07.728343 (+ 211us) client_negotiation.cc:240] Negotiation successful
0924 04:47:07.728805 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":134,"threads_started":1}
I20260924 04:47:07.730819 6689 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42905
I20260924 04:47:07.731117 6689 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:07.731710 6689 heartbeater.cc:511] Master 127.0.98.62:42905 requested a full tablet report, sending...
I20260924 04:47:07.733345 6496 ts_manager.cc:194] Registered new tserver with Master: 3ca550e78f9e4458bfe8316441d9e1af (127.0.98.1:43471)
I20260924 04:47:07.734437 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 04:47:07.742959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42905
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:07.851456 6694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:07.851749 6694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:07.851794 6694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:07.855980 6694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:07.856065 6694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:07.856127 6694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:47:07.861855 6694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:07.863185 6694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:07.864202 6694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:07.874081 6699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:07.874094 6700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:07.874295 6694 server_base.cc:1034] running on GCE node
W20260924 04:47:07.874382 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:07.874946 6694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:07.875676 6694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:07.876879 6694 hybrid_clock.cc:648] HybridClock initialized: now 1790225227876846 us; error 36 us; skew 500 ppm
Sep 24 04:47:07 dist-test-slave-hvcl krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:07.879633 6694 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:47:07.880502 6694 webserver.cc:460] Webserver started at http://127.0.98.2:37593/ using document root <none> and password file <none>
I20260924 04:47:07.881037 6694 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:07.881100 6694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:07.881304 6694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:07.883355 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "4aa7289a60014410bbe464983b693e49"
format_stamp: "Formatted at 2026-09-24 04:47:07 on dist-test-slave-hvcl"
server_key: "a113a2e527646cec9380b0cceb687b0a"
server_key_iv: "53f2bb69365aa688a0bcb29479d9cf1d"
server_key_version: "encryptionkey@0"
I20260924 04:47:07.884009 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4aa7289a60014410bbe464983b693e49"
format_stamp: "Formatted at 2026-09-24 04:47:07 on dist-test-slave-hvcl"
server_key: "a113a2e527646cec9380b0cceb687b0a"
server_key_iv: "53f2bb69365aa688a0bcb29479d9cf1d"
server_key_version: "encryptionkey@0"
I20260924 04:47:07.887820 6694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:47:07.890190 6709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:07.891448 6694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:07.891559 6694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "4aa7289a60014410bbe464983b693e49"
format_stamp: "Formatted at 2026-09-24 04:47:07 on dist-test-slave-hvcl"
server_key: "a113a2e527646cec9380b0cceb687b0a"
server_key_iv: "53f2bb69365aa688a0bcb29479d9cf1d"
server_key_version: "encryptionkey@0"
I20260924 04:47:07.891652 6694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:07.901345 6694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:07.904098 6694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:07.904260 6694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:07.904776 6694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:07.905510 6694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:07.905553 6694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:07.905589 6694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:07.905609 6694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:07.914062 6694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45527
I20260924 04:47:07.914093 6822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45527 every 8 connection(s)
I20260924 04:47:07.914937 6694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:47:07.918953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 6694
I20260924 04:47:07.919080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:47:07.919337 392 external_mini_cluster.cc:1428] Setting key 8b3988cf0d4e46c6b9aa9ae6c1425120
Sep 24 04:47:07 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:47:07.928707 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:07.916713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41527 (local address 127.0.98.62:42905)
0924 04:47:07.916880 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:07.916884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:07.917812 (+ 928us) server_negotiation.cc:407] Connection header received
0924 04:47:07.919149 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:07.919152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:07.919208 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:07.919289 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:07.920596 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:07.921131 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:07.922264 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:07.922484 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:07.922571 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:07.925157 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:07.925160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:07.925162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:07.925185 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:07.926637 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:07.927167 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:07.927169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:07.927171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:07.927244 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:07.927552 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:07.927554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:07.927555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:07.927769 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:07.927854 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:07.928383 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 04:47:07.928541 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 04:47:07.929580 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:07.917173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.2:41527)
0924 04:47:07.917644 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:07.917686 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:07.918909 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:07.919478 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:07.919488 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:07.919848 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:07.920455 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:07.920474 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:07.921249 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:07.921252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:07.922138 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:07.922147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:07.922620 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:07.922624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:07.922802 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:07.923377 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:07.923393 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:07.924995 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:07.926743 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:07.926749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:07.926766 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:07.927071 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:07.927323 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:07.927326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:07.927328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:07.927459 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:07.927862 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:07.927868 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:07.928122 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 04:47:07.928311 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 04:47:07.928644 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260924 04:47:07.930788 6823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42905
I20260924 04:47:07.931061 6823 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:07.931537 6823 heartbeater.cc:511] Master 127.0.98.62:42905 requested a full tablet report, sending...
I20260924 04:47:07.932601 6496 ts_manager.cc:194] Registered new tserver with Master: 4aa7289a60014410bbe464983b693e49 (127.0.98.2:45527)
I20260924 04:47:07.933143 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:47:07.945861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42905
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:08.057463 6828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:08.057772 6828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:08.057835 6828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:08.062001 6828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:08.062063 6828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:08.062143 6828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:47:08.067946 6828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:08.069047 6828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:08.070093 6828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:08.079660 6833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:08.079655 6836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:08.079914 6828 server_base.cc:1034] running on GCE node
W20260924 04:47:08.079654 6834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:08.080356 6828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:08.081008 6828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:08.082167 6828 hybrid_clock.cc:648] HybridClock initialized: now 1790225228082143 us; error 42 us; skew 500 ppm
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:08.084903 6828 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:47:08.085732 6828 webserver.cc:460] Webserver started at http://127.0.98.3:43165/ using document root <none> and password file <none>
I20260924 04:47:08.086292 6828 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:08.086355 6828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:08.086547 6828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:08.088650 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "78f3d9a08a444cc59b0acd1e563b7463"
format_stamp: "Formatted at 2026-09-24 04:47:08 on dist-test-slave-hvcl"
server_key: "84c4ab4c8ac65c56e528f0dbf6eed71e"
server_key_iv: "87fb74485a0c5e52b1e449f80c89e7c6"
server_key_version: "encryptionkey@0"
I20260924 04:47:08.089188 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "78f3d9a08a444cc59b0acd1e563b7463"
format_stamp: "Formatted at 2026-09-24 04:47:08 on dist-test-slave-hvcl"
server_key: "84c4ab4c8ac65c56e528f0dbf6eed71e"
server_key_iv: "87fb74485a0c5e52b1e449f80c89e7c6"
server_key_version: "encryptionkey@0"
I20260924 04:47:08.093142 6828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 04:47:08.095739 6843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:08.097093 6828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 04:47:08.097203 6828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "78f3d9a08a444cc59b0acd1e563b7463"
format_stamp: "Formatted at 2026-09-24 04:47:08 on dist-test-slave-hvcl"
server_key: "84c4ab4c8ac65c56e528f0dbf6eed71e"
server_key_iv: "87fb74485a0c5e52b1e449f80c89e7c6"
server_key_version: "encryptionkey@0"
I20260924 04:47:08.097306 6828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:08.111212 6828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:08.114341 6828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:08.114523 6828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:08.115123 6828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:08.115991 6828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:08.116037 6828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:08.116091 6828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:08.116119 6828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:08.124440 6828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45177
I20260924 04:47:08.124511 6956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45177 every 8 connection(s)
I20260924 04:47:08.125337 6828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:47:08.132194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 6828
I20260924 04:47:08.132318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:47:08.132588 392 external_mini_cluster.cc:1428] Setting key aeee8166a0ec767ccf02daf1dcc4fd34
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:47:08.138871 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.127324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49115 (local address 127.0.98.62:42905)
0924 04:47:08.127468 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.127472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.128299 (+ 827us) server_negotiation.cc:407] Connection header received
0924 04:47:08.129646 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.129650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.129705 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.129776 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.130935 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.131532 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.132954 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.133168 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.133264 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.135630 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.135634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.135636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.135661 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.136977 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.137479 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.137482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.137485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.137569 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.137864 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.137867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.137869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.138038 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.138117 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.138590 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.138716 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:47:08.139654 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.127677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.3:49115)
0924 04:47:08.128143 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.128190 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.129441 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.129948 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.129954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.130316 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.130801 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.130815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.131647 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.131650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.132844 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.132855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.133297 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.133300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.133473 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.134072 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.134087 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.135486 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.137096 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.137102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.137116 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.137387 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.137650 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.137653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.137655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.137784 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.138128 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.138134 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.138375 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.138535 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.138842 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260924 04:47:08.140727 6957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42905
I20260924 04:47:08.140933 6957 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:08.141371 6957 heartbeater.cc:511] Master 127.0.98.62:42905 requested a full tablet report, sending...
I20260924 04:47:08.142293 6496 ts_manager.cc:194] Registered new tserver with Master: 78f3d9a08a444cc59b0acd1e563b7463 (127.0.98.3:45177)
I20260924 04:47:08.142899 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49115
I20260924 04:47:08.146097 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 04:47:08.171794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34359
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42905,127.0.98.61:34359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:08.280997 6962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:08.281246 6962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:08.281284 6962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:08.285251 6962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:08.285310 6962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:08.285327 6962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:08.285341 6962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:08.285355 6962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:47:08.290671 6962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42905,127.0.98.61:34359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
W20260924 04:47:08.290870 6962 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:42905,127.0.98.61:34359'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 04:47:08.291716 6962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:08.292800 6962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:08.301380 6970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:08.301400 6968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:08.301461 6962 server_base.cc:1034] running on GCE node
W20260924 04:47:08.301381 6967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:08.302196 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:08.303453 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:08.304615 6962 hybrid_clock.cc:648] HybridClock initialized: now 1790225228304586 us; error 43 us; skew 500 ppm
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:08.307438 6962 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 04:47:08.308372 6962 webserver.cc:460] Webserver started at http://127.0.98.61:41029/ using document root <none> and password file <none>
I20260924 04:47:08.308903 6962 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:08.308960 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:08.309142 6962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:08.311209 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/data/instance:
uuid: "aab32ac08cfb45928b8ceaeb6eba4d83"
format_stamp: "Formatted at 2026-09-24 04:47:08 on dist-test-slave-hvcl"
server_key: "e872be0f784dd52d011d6fad802bc37b"
server_key_iv: "af4e5adeb6c27912ba566b849f27a815"
server_key_version: "encryptionkey@0"
I20260924 04:47:08.311735 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/wal/instance:
uuid: "aab32ac08cfb45928b8ceaeb6eba4d83"
format_stamp: "Formatted at 2026-09-24 04:47:08 on dist-test-slave-hvcl"
server_key: "e872be0f784dd52d011d6fad802bc37b"
server_key_iv: "af4e5adeb6c27912ba566b849f27a815"
server_key_version: "encryptionkey@0"
I20260924 04:47:08.315339 6962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:47:08.317618 6977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:08.318856 6962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:08.318954 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/wal
uuid: "aab32ac08cfb45928b8ceaeb6eba4d83"
format_stamp: "Formatted at 2026-09-24 04:47:08 on dist-test-slave-hvcl"
server_key: "e872be0f784dd52d011d6fad802bc37b"
server_key_iv: "af4e5adeb6c27912ba566b849f27a815"
server_key_version: "encryptionkey@0"
I20260924 04:47:08.319089 6962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:08.349131 6962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:08.352440 6962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:08.352618 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:08.359515 6962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34359
I20260924 04:47:08.359772 7029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34359 every 8 connection(s)
I20260924 04:47:08.360466 6962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:47:08.363299 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:08.367031 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:47:08.367861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 6962
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:47:08.380218 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.368900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.0.98.62:42905)
0924 04:47:08.369026 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.369031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.370029 (+ 998us) server_negotiation.cc:407] Connection header received
0924 04:47:08.370778 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.370781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.370830 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.370907 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.371801 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.372322 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.373396 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.373616 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.373763 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.376408 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.376412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.376415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.376444 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.377924 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.378533 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.378536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.378538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.378628 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.378944 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.378947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.378948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.379244 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.379331 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.379929 (+ 598us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.380049 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:47:08.381184 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.369263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:48518)
0924 04:47:08.369881 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.369926 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.370558 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.370963 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.370990 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.371413 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.371661 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.371678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.372443 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.372447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.373284 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.373292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.373889 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.373893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.374004 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.374650 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.374665 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.376252 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.378044 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.378051 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.378061 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.378445 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.378716 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.378719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.378721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.378852 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.379334 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.379341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.379690 (+ 349us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.379857 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.380156 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":99,"spinlock_wait_cycles":77440,"thread_start_us":150,"threads_started":1}
I20260924 04:47:08.381184 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.369263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.0.98.61:34359)
0924 04:47:08.369707 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.369725 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.369765 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:47:08.370552 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.370575 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.370812 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.370990 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.371687 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.372646 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.373427 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.373806 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.374019 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.376751 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.376759 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.376772 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.376797 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.378497 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.378936 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.378941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.378943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.379039 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.379283 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.379286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.379288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.379688 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.379905 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.380126 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.380244 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":303,"thread_start_us":139,"threads_started":1}
I20260924 04:47:08.382489 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:47:08.389595 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.383012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.0.98.61:34359)
0924 04:47:08.383079 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.383083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.383122 (+ 39us) server_negotiation.cc:407] Connection header received
0924 04:47:08.383234 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.383237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.383279 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.383337 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.383643 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.384166 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.384875 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.385094 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.385167 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.386883 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.386886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.386889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.386910 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.388192 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.388559 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.388562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.388564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.388626 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.388878 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.388881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.388883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.389043 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.389121 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.389325 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.389468 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":22}
I20260924 04:47:08.389595 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.382915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34359 (local address 127.0.0.1:45214)
0924 04:47:08.383071 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.383084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.383157 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.383362 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.383364 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.383482 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.383567 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.383571 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.384241 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.384243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.384787 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.384794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.385171 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.385173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.385237 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.385609 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.385619 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.386754 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.388311 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.388314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.388316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.388475 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.388699 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.388701 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.388703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.388783 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.389120 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.389123 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.389199 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.389285 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.389418 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 04:47:08.392797 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: Bootstrap starting.
I20260924 04:47:08.395143 7030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:08.395891 7030 log.cc:824] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:08.397895 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: No bootstrap required, opened a new log
I20260924 04:47:08.401226 7030 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } }
I20260924 04:47:08.401439 7030 raft_consensus.cc:374] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:08.401501 7030 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab32ac08cfb45928b8ceaeb6eba4d83, State: Initialized, Role: FOLLOWER
I20260924 04:47:08.401928 7030 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } }
I20260924 04:47:08.402391 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } } }
I20260924 04:47:08.402526 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:47:08.403203 7030 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:47:08.407435 7047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:47:08.407505 7047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:47:08.409997 6962 master_runner.cc:428] Detected that this master aab32ac08cfb45928b8ceaeb6eba4d83 is joining an existing cluster
I20260924 04:47:08.410051 6962 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:34359
I20260924 04:47:08.420516 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.412490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.0.98.62:42905)
0924 04:47:08.412606 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.412609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.412619 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:47:08.412652 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.412654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.412689 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.412756 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.413270 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.413801 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.414948 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.415238 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.415327 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.417213 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.417218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.417220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.417245 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.418883 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.419426 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.419429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.419432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.419513 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.419753 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.419757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.419758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.419937 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.420019 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.420258 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.420362 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:47:08.420814 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.412094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34359 (local address 127.0.0.1:45222)
0924 04:47:08.412981 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.412994 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.413082 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.413369 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.413371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.413578 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.413714 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.413721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.414405 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.414408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.414960 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.414993 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.415480 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.415483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.415559 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.416745 (+ 1186us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.416759 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.417040 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.419156 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.419158 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.419160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.419432 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.419684 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.419687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.419688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.419757 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.420127 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.420129 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.420197 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.420547 (+ 350us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.420646 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":101,"threads_started":1}
I20260924 04:47:08.421033 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.412449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45222 (local address 127.0.98.61:34359)
0924 04:47:08.412551 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.412555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.413069 (+ 514us) server_negotiation.cc:407] Connection header received
0924 04:47:08.413152 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.413155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.413199 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.413277 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.413824 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.414309 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.415073 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.415388 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.415485 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.417172 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.417176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.417179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.417202 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.419051 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.419522 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.419525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.419527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.419591 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.419834 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.419837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.419839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.420016 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.420076 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.420333 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.420435 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:47:08.420527 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.412077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:48526)
0924 04:47:08.412360 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.412373 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.412463 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.412791 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.412794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.413032 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.413166 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.413172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.413887 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.413890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.414827 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.414837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.415345 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.415348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.415426 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.416123 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.416139 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.416602 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.419039 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.419042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.419044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.419338 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.419591 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.419594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.419596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.419674 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.420020 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.420023 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.420108 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.420190 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.420336 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":108,"threads_started":1}
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:47:08.436309 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.425875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34359 (local address 127.0.0.1:45224)
0924 04:47:08.426259 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.426315 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.426480 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.426875 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.426881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.427365 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.427522 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.427531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.428284 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.428288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.429317 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.429328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.429848 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.429851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.430000 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.430565 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.430580 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.432444 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.434370 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.434380 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.434387 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.434717 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.435048 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.435051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.435053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.435138 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.435612 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.435620 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.435770 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.435915 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.436137 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":114,"threads_started":1}
I20260924 04:47:08.436841 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.425844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.0.98.61:34359)
0924 04:47:08.425957 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.425961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.426383 (+ 422us) server_negotiation.cc:407] Connection header received
0924 04:47:08.426660 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.426663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.426760 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.426831 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.427670 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.428182 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.429457 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.429680 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.429831 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.432620 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.432624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.432627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.432655 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.434258 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.434829 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.434832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.434834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.434917 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.435231 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.435234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.435236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.435456 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.435542 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.436566 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.436689 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:47:08.439940 6496 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:34359: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 34359
}
attrs {
promote: true
}
}
dest_uuid: "362ebf6fc3494b56afca08f7ea649efb"
cas_config_opid_index: -1
I20260924 04:47:08.440613 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } }
I20260924 04:47:08.443092 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER aab32ac08cfb45928b8ceaeb6eba4d83 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } } }
I20260924 04:47:08.443373 7004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:08.444741 7004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 FOLLOWER]: Refusing update from remote peer 362ebf6fc3494b56afca08f7ea649efb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 04:47:08.444861 7058 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:34359
I20260924 04:47:08.445166 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } } }
I20260924 04:47:08.445263 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:08.445672 7058 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:08.447201 6962 master.cc:537] Master@127.0.98.61:34359 shutting down...
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260924 04:47:08.453858 7004 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER aab32ac08cfb45928b8ceaeb6eba4d83 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } } }
I20260924 04:47:08.457160 7067 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } } }
I20260924 04:47:08.457372 7067 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:47:08.457604 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 362ebf6fc3494b56afca08f7ea649efb. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } } }
I20260924 04:47:08.457690 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:47:08.457855 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } } }
I20260924 04:47:08.457919 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:47:08.458570 6962 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 LEARNER]: Raft consensus shutting down.
I20260924 04:47:08.458851 6962 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 LEARNER]: Raft consensus is shut down!
I20260924 04:47:08.458928 6962 tablet_replica.cc:330] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 04:47:08.474385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38771
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42905,127.0.98.61:34359,127.0.98.60:38771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:47:08.474705 6962 master.cc:552] Master@127.0.98.61:34359 shutdown complete.
I20260924 04:47:08.474769 6962 master_runner.cc:305] Clearing existing system tablet
I20260924 04:47:08.475042 6962 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:08.475091 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:08.476320 6962 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:08.477046 7070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:08.477252 6962 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:08.477368 6962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:08.477403 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/wal
uuid: "aab32ac08cfb45928b8ceaeb6eba4d83"
format_stamp: "Formatted at 2026-09-24 04:47:08 on dist-test-slave-hvcl"
server_key: "e872be0f784dd52d011d6fad802bc37b"
server_key_iv: "af4e5adeb6c27912ba566b849f27a815"
server_key_version: "encryptionkey@0"
I20260924 04:47:08.478902 6962 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:08.480815 6962 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 04:47:08.480890 6962 log.cc:1196] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 04:47:08.481120 6962 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: Deleting consensus metadata
I20260924 04:47:08.481268 6962 master_runner.cc:315] Copying system tablet from 127.0.98.62:42905
I20260924 04:47:08.482262 6962 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42905
I20260924 04:47:08.489537 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.482855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.0.98.62:42905)
0924 04:47:08.483043 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.483048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.483199 (+ 151us) server_negotiation.cc:407] Connection header received
0924 04:47:08.483329 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.483332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.483378 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.483441 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.483865 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.484398 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.485187 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.485417 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.485501 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.486556 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.486560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.486580 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.486607 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.488035 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.488450 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.488453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.488455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.488532 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.488772 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.488775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.488777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.488946 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.489035 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.489275 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.489376 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:47:08.489540 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.482772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:48540)
0924 04:47:08.483118 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.483130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.483240 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.483466 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.483468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.483658 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.483758 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.483764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.484493 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.484497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.485080 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.485089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.485513 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.485516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.485603 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.486069 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.486084 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.486434 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.488128 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.488131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.488133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.488326 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.488616 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.488618 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.488619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.488684 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.489037 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.489041 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.489130 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.489212 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.489356 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":75,"threads_started":1}
I20260924 04:47:08.490346 6518 tablet_copy_service.cc:140] P 362ebf6fc3494b56afca08f7ea649efb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer aab32ac08cfb45928b8ceaeb6eba4d83 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48540)
I20260924 04:47:08.490502 6518 tablet_copy_service.cc:161] P 362ebf6fc3494b56afca08f7ea649efb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer aab32ac08cfb45928b8ceaeb6eba4d83 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48540: session id = aab32ac08cfb45928b8ceaeb6eba4d83-00000000000000000000000000000000
I20260924 04:47:08.491942 6518 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 04:47:08.493450 6962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 04:47:08.495498 6962 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: tablet copy: Starting download of 0 data blocks...
I20260924 04:47:08.495685 6962 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: tablet copy: Starting download of 1 WAL segments...
I20260924 04:47:08.497926 6962 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:47:08.499647 6518 tablet_copy_service.cc:342] P 362ebf6fc3494b56afca08f7ea649efb: Request end of tablet copy session aab32ac08cfb45928b8ceaeb6eba4d83-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48540
I20260924 04:47:08.499765 6518 tablet_copy_service.cc:433] P 362ebf6fc3494b56afca08f7ea649efb: ending tablet copy session aab32ac08cfb45928b8ceaeb6eba4d83-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer aab32ac08cfb45928b8ceaeb6eba4d83
I20260924 04:47:08.507144 6962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:08.508987 7077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:08.509097 7080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:08.509260 7078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:08.509339 6962 server_base.cc:1034] running on GCE node
I20260924 04:47:08.509615 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:08.510003 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:08.511126 6962 hybrid_clock.cc:648] HybridClock initialized: now 1790225228511119 us; error 26 us; skew 500 ppm
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:08.512542 6962 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 04:47:08.512957 6962 webserver.cc:460] Webserver started at http://127.0.98.61:34723/ using document root <none> and password file <none>
I20260924 04:47:08.513134 6962 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:08.513170 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:08.514174 6962 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:08.514963 7087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:08.515192 6962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:08.515257 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/wal
uuid: "aab32ac08cfb45928b8ceaeb6eba4d83"
format_stamp: "Formatted at 2026-09-24 04:47:08 on dist-test-slave-hvcl"
server_key: "e872be0f784dd52d011d6fad802bc37b"
server_key_iv: "af4e5adeb6c27912ba566b849f27a815"
server_key_version: "encryptionkey@0"
I20260924 04:47:08.515334 6962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:08.528951 6962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:08.530764 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:08.535156 6962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34359
I20260924 04:47:08.535180 7139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34359 every 8 connection(s)
I20260924 04:47:08.535657 6962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:47:08.537000 7140 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:47:08.537798 7140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: Bootstrap starting.
I20260924 04:47:08.544168 7140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:47:08.544450 7140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: Bootstrap complete.
I20260924 04:47:08.544921 7140 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } }
I20260924 04:47:08.545029 7140 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aab32ac08cfb45928b8ceaeb6eba4d83, State: Initialized, Role: LEARNER
I20260924 04:47:08.545128 7140 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } }
I20260924 04:47:08.545428 7143 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } } }
I20260924 04:47:08.545564 7140 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:08.545595 7143 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:47:08.548704 7154 catalog_manager.cc:1259] Loaded cluster ID: 2723da5504bc4f1a893a21ee44f47e03
I20260924 04:47:08.548763 7154 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: loading cluster ID for follower catalog manager: success
I20260924 04:47:08.550330 7154 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: acquiring CA information for follower catalog manager: success
I20260924 04:47:08.551434 7154 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 04:47:08.620795 7068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:08.621061 7068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:08.621101 7068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:08.625384 7068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:08.625465 7068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:08.625485 7068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:08.625501 7068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:08.625515 7068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:47:08.631161 7068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42905,127.0.98.61:34359,127.0.98.60:38771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:08.632237 7068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:08.633276 7068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:08.642185 7160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:08.642220 7159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:08.642340 7068 server_base.cc:1034] running on GCE node
W20260924 04:47:08.642216 7162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:08.643151 7068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:08.644323 7068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:08.645473 7068 hybrid_clock.cc:648] HybridClock initialized: now 1790225228645452 us; error 37 us; skew 500 ppm
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:08.648609 7068 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 04:47:08.649580 7068 webserver.cc:460] Webserver started at http://127.0.98.60:35059/ using document root <none> and password file <none>
I20260924 04:47:08.650151 7068 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:08.650221 7068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:08.650411 7068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:08.652558 7068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/data/instance:
uuid: "211a15b5a8e943198a6ad0b45931d1de"
format_stamp: "Formatted at 2026-09-24 04:47:08 on dist-test-slave-hvcl"
server_key: "0941602ff64a2576dd08de8e2821bd1f"
server_key_iv: "9afcdb2865fc5be183dde64cbfbb1896"
server_key_version: "encryptionkey@0"
I20260924 04:47:08.653106 7068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/wal/instance:
uuid: "211a15b5a8e943198a6ad0b45931d1de"
format_stamp: "Formatted at 2026-09-24 04:47:08 on dist-test-slave-hvcl"
server_key: "0941602ff64a2576dd08de8e2821bd1f"
server_key_iv: "9afcdb2865fc5be183dde64cbfbb1896"
server_key_version: "encryptionkey@0"
I20260924 04:47:08.657121 7068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 04:47:08.659529 7169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:08.660846 7068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:08.660969 7068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/wal
uuid: "211a15b5a8e943198a6ad0b45931d1de"
format_stamp: "Formatted at 2026-09-24 04:47:08 on dist-test-slave-hvcl"
server_key: "0941602ff64a2576dd08de8e2821bd1f"
server_key_iv: "9afcdb2865fc5be183dde64cbfbb1896"
server_key_version: "encryptionkey@0"
I20260924 04:47:08.661077 7068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:08.677917 7068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:08.681012 7068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:08.681159 7068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:08.688819 7068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38771
I20260924 04:47:08.689057 7221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38771 every 8 connection(s)
I20260924 04:47:08.689764 7068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:47:08.691654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 7068
I20260924 04:47:08.692881 7222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:08.697425 7222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:47:08.707221 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.693183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32788 (local address 127.0.98.60:38771)
0924 04:47:08.693569 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.693588 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.693631 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:47:08.694462 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.694485 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.694917 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.695316 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.696134 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.697562 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.698704 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.699329 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.699484 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.702249 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.702254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.702263 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.702289 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.704318 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.704776 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.704781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.704788 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.704929 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.705175 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.705178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.705179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.705586 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.705739 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.706059 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.706306 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":110,"threads_started":1}
I20260924 04:47:08.708741 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.699300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:48554)
0924 04:47:08.699821 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.699868 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.700023 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.700321 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.700326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.700658 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.700801 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.700807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.701857 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.701861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.702730 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.702741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.703345 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.703349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.703510 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.703962 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.703974 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.705521 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.707270 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.707277 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.707283 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.707547 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.707810 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.707813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.707815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.707904 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.708202 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.708208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.708322 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.708460 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.708599 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":187,"threads_started":1}
I20260924 04:47:08.708835 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.699453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.0.98.62:42905)
0924 04:47:08.699584 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.699588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.699939 (+ 351us) server_negotiation.cc:407] Connection header received
0924 04:47:08.700143 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.700146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.700194 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.700259 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.700978 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.701734 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.702838 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.703194 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.703297 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.705649 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.705653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.705656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.705684 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.707166 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.707640 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.707643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.707645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.707729 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.707980 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.707982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.707984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.708127 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.708193 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.708560 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.708682 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:47:08.710098 7222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:47:08.717999 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.710523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34359 (local address 127.0.0.1:45226)
0924 04:47:08.710657 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.710672 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.710744 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.711152 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.711154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.711281 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.711376 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.711381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.712183 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.712185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.712740 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.712747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.713369 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.713372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.713441 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.713836 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.713849 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.714947 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.716812 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.716814 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.716816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.716995 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.717253 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.717255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.717256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.717331 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.717626 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.717628 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.717705 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.717786 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.717889 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 04:47:08.718050 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.710658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.0.98.61:34359)
0924 04:47:08.710930 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.710934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.710945 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:47:08.711007 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.711010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.711053 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.711124 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.711458 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.712105 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.712867 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.713242 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.713356 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.715102 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.715106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.715108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.715135 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.716705 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.717079 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.717082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.717083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.717153 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.717399 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.717401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.717403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.717560 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.717627 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.717837 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.717933 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":110,"threads_started":1}
I20260924 04:47:08.718653 7222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:47:08.725944 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.719091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38771 (local address 127.0.0.1:32790)
0924 04:47:08.719193 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.719212 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.719304 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.719483 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.719486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.719614 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.719671 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.719676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.720857 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.720861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.721344 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.721350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.721735 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.721738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.721802 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.722182 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.722192 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.723303 (+ 1111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.724704 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.724707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.724709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.724876 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.725097 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.725100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.725102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.725165 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.725474 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.725477 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.725549 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.725627 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.725750 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 04:47:08.725944 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.719145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.0.98.60:38771)
0924 04:47:08.719235 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.719239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.719282 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:47:08.719372 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.719374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.719409 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.719465 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.719798 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.720763 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.721435 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.721652 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.721739 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.723409 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.723413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.723415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.723436 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.724606 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.724957 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.724960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.724962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.725028 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.725233 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.725236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.725238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.725396 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.725473 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.725675 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.725803 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":36}
I20260924 04:47:08.728772 7222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: Bootstrap starting.
I20260924 04:47:08.730917 7222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:08.731622 7222 log.cc:824] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:08.733467 7222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: No bootstrap required, opened a new log
I20260924 04:47:08.736639 6689 heartbeater.cc:503] Master 127.0.98.62:42905 was elected leader, sending a full tablet report...
I20260924 04:47:08.736645 7222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } }
I20260924 04:47:08.736907 7222 raft_consensus.cc:374] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:08.736963 7222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 211a15b5a8e943198a6ad0b45931d1de, State: Initialized, Role: FOLLOWER
I20260924 04:47:08.737416 7222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } }
I20260924 04:47:08.737846 7228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } } }
I20260924 04:47:08.737998 7228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:47:08.738617 7222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:47:08.743599 7239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:47:08.743686 7239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:47:08.746376 7068 master_runner.cc:428] Detected that this master 211a15b5a8e943198a6ad0b45931d1de is joining an existing cluster
I20260924 04:47:08.746428 7068 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:38771
I20260924 04:47:08.757311 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.748929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.0.98.62:42905)
0924 04:47:08.749044 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.749050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.749438 (+ 388us) server_negotiation.cc:407] Connection header received
0924 04:47:08.749605 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.749608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.749655 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.749705 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.750328 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.751189 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.752470 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.752802 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.752875 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.754066 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.754069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.754071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.754094 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.755637 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.756190 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.756193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.756194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.756272 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.756546 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.756549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.756551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.756735 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.756858 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.757099 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.757192 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:47:08.757632 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.748935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:48566)
0924 04:47:08.749350 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.749362 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.749503 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.749840 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.749842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.750054 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.750186 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.750193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.751318 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.751321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.752331 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.752341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.753023 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.753026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.753115 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.753515 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.753529 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.753925 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.755725 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.755728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.755731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.756037 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.756349 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.756352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.756354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.756444 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.756854 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.756857 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.756947 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.757047 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.757174 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":36,"thread_start_us":88,"threads_started":1}
I20260924 04:47:08.760751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 6560
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:47:08.768009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43471
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42905
--webserver_port=39941
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42905,127.0.98.61:34359
--tserver_master_addrs=127.0.98.62:42905,127.0.98.61:34359,127.0.98.60:38771 with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:47:08.775065 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.748554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.0.98.60:38771)
0924 04:47:08.748768 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.748772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.749354 (+ 582us) server_negotiation.cc:407] Connection header received
0924 04:47:08.749480 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.749483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.749515 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.749556 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.750074 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.750770 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.751930 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.752357 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.752436 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.771691 (+ 19255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.771695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.771697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.771716 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.773197 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.773625 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.773628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.773630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.773735 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.774000 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.774003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.774005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.774265 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.774334 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.774637 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.774871 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":146}
I20260924 04:47:08.775554 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.748482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38771 (local address 127.0.0.1:32796)
0924 04:47:08.749290 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.749302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.749404 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.749675 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.749677 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.749872 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.749981 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.749987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.750872 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.750875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.751820 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.751829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.752544 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.752548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.752626 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.753151 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.753165 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.771551 (+ 18386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.773303 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.773306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.773308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.773535 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.773832 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.773835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.773837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.773916 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.774402 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.774405 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.774487 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.775255 (+ 768us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.775390 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":32,"thread_start_us":110,"threads_started":1}
I20260924 04:47:08.776721 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.748467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34359 (local address 127.0.0.1:45228)
0924 04:47:08.748960 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.748974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.749070 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.749401 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.749404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.749632 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.749763 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.749770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.750850 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.750853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.752241 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.752247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.752888 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.752891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.752976 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.771634 (+ 18658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.771649 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.772607 (+ 958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.774496 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.774499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.774501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.774704 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.775018 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.775021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.775023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.775100 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.776234 (+ 1134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.776238 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.776325 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.776433 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.776562 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":89,"thread_start_us":109,"threads_started":1}
I20260924 04:47:08.777206 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.748653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45228 (local address 127.0.98.61:34359)
0924 04:47:08.748819 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.748823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.749032 (+ 209us) server_negotiation.cc:407] Connection header received
0924 04:47:08.749146 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.749149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.749202 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.749273 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.749891 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.750722 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.752348 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.752662 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.752733 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.772796 (+ 20063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.772801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.772804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.772839 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.774396 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.774804 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.774807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.774809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.774898 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.775412 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.775416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.775417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.775645 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.775732 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.776991 (+ 1259us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.777102 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 04:47:08.780534 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.761433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38771 (local address 127.0.0.1:32798)
0924 04:47:08.761529 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.761543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.761612 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.763669 (+ 2057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.763672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.763871 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.763966 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.763973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.765135 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.765139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.765749 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.765758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.766222 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.766225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.766317 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.766730 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.766744 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.767962 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.779175 (+ 11213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.779179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.779181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.779397 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.779657 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.779661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.779663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.779746 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.780144 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.780147 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.780234 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.780295 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.780419 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 04:47:08.781116 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.763162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.0.98.60:38771)
0924 04:47:08.763387 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.763394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.763407 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:47:08.763471 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.763474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.763522 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.763641 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.764085 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.764998 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.765898 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.766112 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.766190 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.768106 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.768111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.768114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.768142 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.779052 (+ 10910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.779484 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.779487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.779489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.779574 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.779825 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.779828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.779830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.780022 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.780839 (+ 817us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.780920 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.781007 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
I20260924 04:47:08.781917 6496 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:38771: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 38771
}
attrs {
promote: true
}
}
dest_uuid: "362ebf6fc3494b56afca08f7ea649efb"
cas_config_opid_index: 5
I20260924 04:47:08.782469 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } }
I20260924 04:47:08.783528 7057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 211a15b5a8e943198a6ad0b45931d1de (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } } }
I20260924 04:47:08.784844 7057 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:38771
I20260924 04:47:08.785038 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } } }
I20260924 04:47:08.785123 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:08.786227 7196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:08.787657 7196 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 1 FOLLOWER]: Refusing update from remote peer 362ebf6fc3494b56afca08f7ea649efb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 04:47:08.788337 7068 master.cc:537] Master@127.0.98.60:38771 shutting down...
I20260924 04:47:08.788987 7058 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:08.789476 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.783660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34359 (local address 127.0.0.1:45230)
0924 04:47:08.784062 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.784076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.784145 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.785521 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.785524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.785541 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:08.785629 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.785635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.786623 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.786627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.787853 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.787864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.789047 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.789051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.789142 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.789188 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.789256 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.789336 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329}
I20260924 04:47:08.789989 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.783869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45230 (local address 127.0.98.61:34359)
0924 04:47:08.785282 (+ 1413us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.785287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.785300 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:47:08.785342 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.785344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.785391 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.785485 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:08.785739 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.786457 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.788061 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.788671 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.788770 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.788934 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.789790 (+ 856us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.789875 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1333}
W20260924 04:47:08.790367 6483 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 04:47:08.790886 7114 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 LEARNER]: Refusing update from remote peer 362ebf6fc3494b56afca08f7ea649efb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260924 04:47:08.791298 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb -> Peer 211a15b5a8e943198a6ad0b45931d1de (127.0.98.60:38771): Couldn't send request to peer 211a15b5a8e943198a6ad0b45931d1de. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:47:08.791843 7058 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 04:47:08.792893 7114 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 211a15b5a8e943198a6ad0b45931d1de (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } } }
I20260924 04:47:08.794190 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 362ebf6fc3494b56afca08f7ea649efb. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } } }
I20260924 04:47:08.794384 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:47:08.794210 7143 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: true } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } } }
I20260924 04:47:08.794497 7143 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:47:08.796541 7068 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:08.796815 7068 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:08.796921 7068 tablet_replica.cc:330] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: stopping tablet replica
I20260924 04:47:08.797144 7058 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb: attempting to promote NON_VOTER aab32ac08cfb45928b8ceaeb6eba4d83 to VOTER
I20260924 04:47:08.797569 7058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } }
I20260924 04:47:08.798889 7114 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 LEARNER]: Refusing update from remote peer 362ebf6fc3494b56afca08f7ea649efb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 04:47:08.799360 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 04:47:08.799834 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb -> Peer 211a15b5a8e943198a6ad0b45931d1de (127.0.98.60:38771): Couldn't send request to peer 211a15b5a8e943198a6ad0b45931d1de. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:47:08.801622 7057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, aab32ac08cfb45928b8ceaeb6eba4d83 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } } }
I20260924 04:47:08.803308 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } } }
I20260924 04:47:08.803419 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:08.803706 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } } }
I20260924 04:47:08.803817 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:08.804133 7114 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, aab32ac08cfb45928b8ceaeb6eba4d83 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } } }
I20260924 04:47:08.810798 7143 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } } }
I20260924 04:47:08.811054 7143 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:47:08.813531 7068 master.cc:552] Master@127.0.98.60:38771 shutdown complete.
I20260924 04:47:08.813585 7068 master_runner.cc:305] Clearing existing system tablet
I20260924 04:47:08.813793 7068 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:08.813843 7068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:08.814934 7068 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:47:08.815766 7256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:08.815933 7068 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:08.816046 7068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:08.816083 7068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/wal
uuid: "211a15b5a8e943198a6ad0b45931d1de"
format_stamp: "Formatted at 2026-09-24 04:47:08 on dist-test-slave-hvcl"
server_key: "0941602ff64a2576dd08de8e2821bd1f"
server_key_iv: "9afcdb2865fc5be183dde64cbfbb1896"
server_key_version: "encryptionkey@0"
I20260924 04:47:08.818058 7068 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:08.819998 7068 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 04:47:08.820081 7068 log.cc:1196] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 04:47:08.820315 7068 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: Deleting consensus metadata
I20260924 04:47:08.820464 7068 master_runner.cc:315] Copying system tablet from 127.0.98.62:42905
I20260924 04:47:08.821468 7068 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42905
I20260924 04:47:08.828713 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.822065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.0.98.62:42905)
0924 04:47:08.822192 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:08.822196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:08.822448 (+ 252us) server_negotiation.cc:407] Connection header received
0924 04:47:08.822559 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:08.822562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:08.822609 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:08.822676 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:08.823135 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.823678 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.824479 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.824701 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.824796 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.825840 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:08.825844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:08.825847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:08.825872 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:08.827237 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.827612 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.827615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.827617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.827691 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:08.827942 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:08.827946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:08.827947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:08.828121 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:08.828201 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:08.828451 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 04:47:08.828555 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:47:08.828727 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:08.821987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:48572)
0924 04:47:08.822358 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:08.822369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:08.822478 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:08.822702 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:08.822705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:08.822896 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:08.823022 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.823030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.823783 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.823786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.824369 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:08.824378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:08.824802 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:08.824805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:08.824895 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:08.825372 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:08.825387 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:08.825694 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:08.827331 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.827334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.827336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.827529 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.827783 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:08.827786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:08.827788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:08.827866 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:08.828203 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:08.828206 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:08.828307 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 04:47:08.828390 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 04:47:08.828540 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":79,"threads_started":1}
I20260924 04:47:08.829466 6518 tablet_copy_service.cc:140] P 362ebf6fc3494b56afca08f7ea649efb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 211a15b5a8e943198a6ad0b45931d1de ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48572)
I20260924 04:47:08.829553 6518 tablet_copy_service.cc:161] P 362ebf6fc3494b56afca08f7ea649efb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 211a15b5a8e943198a6ad0b45931d1de at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48572: session id = 211a15b5a8e943198a6ad0b45931d1de-00000000000000000000000000000000
I20260924 04:47:08.830363 6518 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 04:47:08.831486 7068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 04:47:08.833341 7068 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: tablet copy: Starting download of 0 data blocks...
I20260924 04:47:08.833508 7068 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: tablet copy: Starting download of 1 WAL segments...
I20260924 04:47:08.834750 7068 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:47:08.836241 6518 tablet_copy_service.cc:342] P 362ebf6fc3494b56afca08f7ea649efb: Request end of tablet copy session 211a15b5a8e943198a6ad0b45931d1de-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48572
I20260924 04:47:08.836319 6518 tablet_copy_service.cc:433] P 362ebf6fc3494b56afca08f7ea649efb: ending tablet copy session 211a15b5a8e943198a6ad0b45931d1de-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 211a15b5a8e943198a6ad0b45931d1de
I20260924 04:47:08.843183 7068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:08.845033 7263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:08.845000 7266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:08.845217 7264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:08.851178 7068 server_base.cc:1034] running on GCE node
I20260924 04:47:08.851387 7068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:08.851699 7068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:08.852877 7068 hybrid_clock.cc:648] HybridClock initialized: now 1790225228852864 us; error 32 us; skew 500 ppm
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:08.854306 7068 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 04:47:08.854779 7068 webserver.cc:460] Webserver started at http://127.0.98.60:36805/ using document root <none> and password file <none>
I20260924 04:47:08.854998 7068 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:08.855041 7068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:08.856181 7068 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:08.856993 7273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:08.857191 7068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:08.857259 7068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/wal
uuid: "211a15b5a8e943198a6ad0b45931d1de"
format_stamp: "Formatted at 2026-09-24 04:47:08 on dist-test-slave-hvcl"
server_key: "0941602ff64a2576dd08de8e2821bd1f"
server_key_iv: "9afcdb2865fc5be183dde64cbfbb1896"
server_key_version: "encryptionkey@0"
I20260924 04:47:08.857348 7068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:08.862133 7068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:08.863914 7068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:08.868249 7068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38771
I20260924 04:47:08.868855 7068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:47:08.870491 7326 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:47:08.871124 7325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38771 every 8 connection(s)
I20260924 04:47:08.871459 7326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: Bootstrap starting.
I20260924 04:47:08.879985 7326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:47:08.880247 7326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: Bootstrap complete.
I20260924 04:47:08.880724 7326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } }
I20260924 04:47:08.880847 7326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 211a15b5a8e943198a6ad0b45931d1de, State: Initialized, Role: LEARNER
I20260924 04:47:08.880947 7326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } }
I20260924 04:47:08.881251 7329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } } }
I20260924 04:47:08.881414 7326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:08.881428 7329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [sys.catalog]: This master's current role is: LEARNER
I20260924 04:47:08.884338 7340 catalog_manager.cc:1259] Loaded cluster ID: 2723da5504bc4f1a893a21ee44f47e03
I20260924 04:47:08.884440 7340 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: loading cluster ID for follower catalog manager: success
I20260924 04:47:08.886145 7340 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: acquiring CA information for follower catalog manager: success
I20260924 04:47:08.887524 7340 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 04:47:08.927253 7249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:08.927590 7249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:08.927659 7249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:08.934038 7249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:08.934130 7249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:08.934223 7249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:47:08.935580 6823 heartbeater.cc:503] Master 127.0.98.62:42905 was elected leader, sending a full tablet report...
I20260924 04:47:08.941462 7249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39941
--tserver_master_addrs=127.0.98.62:42905,127.0.98.61:34359,127.0.98.60:38771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:08.942515 7249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:08.943524 7249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:08.954077 7345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:08.954175 7346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:08.954094 7348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:08.954144 7249 server_base.cc:1034] running on GCE node
I20260924 04:47:08.954708 7249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:08.955442 7249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:08.956602 7249 hybrid_clock.cc:648] HybridClock initialized: now 1790225228956575 us; error 49 us; skew 500 ppm
Sep 24 04:47:08 dist-test-slave-hvcl krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:08.959420 7249 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:47:08.960278 7249 webserver.cc:460] Webserver started at http://127.0.98.1:39941/ using document root <none> and password file <none>
I20260924 04:47:08.960820 7249 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:08.960899 7249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:08.964799 7249 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:47:08.966945 7355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:08.968254 7249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:08.968364 7249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "3ca550e78f9e4458bfe8316441d9e1af"
format_stamp: "Formatted at 2026-09-24 04:47:07 on dist-test-slave-hvcl"
server_key: "99ddda11c3f7ae9f6613d2949e734820"
server_key_iv: "fb65d00dcfbde9a3e6f5631ba141604a"
server_key_version: "encryptionkey@0"
I20260924 04:47:08.968945 7249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:09.008747 7249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:09.011963 7249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:09.012142 7249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:09.012728 7249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:09.013563 7249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:09.013610 7249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:09.013662 7249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:09.013688 7249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:09.022792 7249 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43471
I20260924 04:47:09.022842 7468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43471 every 8 connection(s)
I20260924 04:47:09.023808 7249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:47:09.025409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 7249
I20260924 04:47:09.025601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 6694
I20260924 04:47:09.032729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45527
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42905
--webserver_port=37593
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42905,127.0.98.61:34359
--tserver_master_addrs=127.0.98.62:42905,127.0.98.61:34359,127.0.98.60:38771 with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:47:09 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:47:09 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:47:09.040189 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.025645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53701 (local address 127.0.98.62:42905)
0924 04:47:09.025801 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.025806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.027469 (+ 1663us) server_negotiation.cc:407] Connection header received
0924 04:47:09.028337 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.028341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.028398 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.028480 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.029997 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.030837 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.032371 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.032707 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.032777 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.036341 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.036344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.036348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.036374 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.037754 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.038453 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.038457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.038458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.038532 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.038907 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.038910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.038912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.039173 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.039290 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.039880 (+ 590us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.040011 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 04:47:09.040520 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.026997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60099 (local address 127.0.98.60:38771)
0924 04:47:09.027290 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.027295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.031388 (+ 4093us) server_negotiation.cc:407] Connection header received
0924 04:47:09.031488 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.031491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.031547 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.032135 (+ 588us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.032242 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.033215 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.034287 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.034634 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.035114 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.036902 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.036906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.036908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.037028 (+ 120us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.038609 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.039015 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.039018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.039020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.039096 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.039350 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.039353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.039355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.039552 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.040153 (+ 601us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.040231 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.040357 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":84,"threads_started":1}
I20260924 04:47:09.041496 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.026180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38771 (local address 127.0.98.1:60099)
0924 04:47:09.031287 (+ 5107us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.031302 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.031404 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.031685 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.031688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.031899 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:09.032029 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.032036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.033327 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.033330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.034162 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.034171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.034768 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.034772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.034854 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.035348 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:09.035362 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:09.036751 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:09.038705 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.038708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.038710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.038910 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.039172 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.039175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.039177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.039258 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.039658 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:09.039661 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:09.039732 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.039801 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.040463 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5035,"mutex_wait_us":134,"thread_start_us":125,"threads_started":1}
I20260924 04:47:09.041895 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.026040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.1:53701)
0924 04:47:09.027320 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.027360 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.028140 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.028662 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.028669 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.029116 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:09.029816 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.029837 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.031045 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.031047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.032250 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.032257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.033217 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.033221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.033416 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.034133 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:09.034153 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:09.036175 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:09.038022 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.038026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.038039 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.038347 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.038610 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.038613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.038615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.038792 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.039284 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:09.039290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:09.039601 (+ 311us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.039800 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.041778 (+ 1978us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"mutex_wait_us":85,"thread_start_us":133,"threads_started":1}
I20260924 04:47:09.042920 7470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42905
I20260924 04:47:09.042953 7469 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38771
I20260924 04:47:09.043190 7469 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:09.043188 7470 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:09.043766 7470 heartbeater.cc:511] Master 127.0.98.62:42905 requested a full tablet report, sending...
I20260924 04:47:09.043768 7469 heartbeater.cc:511] Master 127.0.98.60:38771 requested a full tablet report, sending...
I20260924 04:47:09.045092 6496 ts_manager.cc:194] Re-registered known tserver with Master: 3ca550e78f9e4458bfe8316441d9e1af (127.0.98.1:43471)
I20260924 04:47:09.045346 7290 ts_manager.cc:194] Registered new tserver with Master: 3ca550e78f9e4458bfe8316441d9e1af (127.0.98.1:43471)
I20260924 04:47:09.045768 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53701
Sep 24 04:47:09 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:47:09.050159 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.026160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49949 (local address 127.0.98.61:34359)
0924 04:47:09.026311 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.026316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.031700 (+ 5384us) server_negotiation.cc:407] Connection header received
0924 04:47:09.031872 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.031876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.031934 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.031984 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.032812 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.033480 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.034547 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.034876 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.034947 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.047192 (+ 12245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.047197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.047200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.047228 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.048632 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.049124 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.049127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.049128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.049210 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.049458 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.049460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.049461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.049623 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.049692 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.049936 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.050035 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:47:09.050226 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.026958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34359 (local address 127.0.98.1:49949)
0924 04:47:09.031619 (+ 4661us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.031631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.031774 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.032115 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.032117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.032533 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:09.032663 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.032670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.033609 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.033613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.034421 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.034431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.035364 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.035366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.035430 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.045332 (+ 9902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:09.045347 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:09.047018 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:09.048741 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.048745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.048748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.048995 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.049296 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.049299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.049301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.049386 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.049694 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:09.049697 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:09.049796 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.049894 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.050048 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4593,"thread_start_us":93,"threads_started":1}
I20260924 04:47:09.050915 7471 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34359
I20260924 04:47:09.051038 7471 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:09.051226 7471 heartbeater.cc:511] Master 127.0.98.61:34359 requested a full tablet report, sending...
I20260924 04:47:09.052382 7104 ts_manager.cc:194] Registered new tserver with Master: 3ca550e78f9e4458bfe8316441d9e1af (127.0.98.1:43471)
I20260924 04:47:09.144966 6957 heartbeater.cc:503] Master 127.0.98.62:42905 was elected leader, sending a full tablet report...
W20260924 04:47:09.150408 7477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:09.150650 7477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:09.150691 7477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:09.154750 7477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:09.154809 7477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:09.154892 7477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:47:09.160408 7477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37593
--tserver_master_addrs=127.0.98.62:42905,127.0.98.61:34359,127.0.98.60:38771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:09.161531 7477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:09.162622 7477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:09.172286 7482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:09.172449 7477 server_base.cc:1034] running on GCE node
W20260924 04:47:09.172596 7483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:09.172282 7485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:09.173135 7477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:09.173815 7477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:09.175016 7477 hybrid_clock.cc:648] HybridClock initialized: now 1790225229174985 us; error 46 us; skew 500 ppm
Sep 24 04:47:09 dist-test-slave-hvcl krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:09.177677 7477 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:47:09.178519 7477 webserver.cc:460] Webserver started at http://127.0.98.2:37593/ using document root <none> and password file <none>
I20260924 04:47:09.179086 7477 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:09.179171 7477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:09.183058 7477 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:47:09.185066 7492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:09.186389 7477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:09.186498 7477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "4aa7289a60014410bbe464983b693e49"
format_stamp: "Formatted at 2026-09-24 04:47:07 on dist-test-slave-hvcl"
server_key: "a113a2e527646cec9380b0cceb687b0a"
server_key_iv: "53f2bb69365aa688a0bcb29479d9cf1d"
server_key_version: "encryptionkey@0"
I20260924 04:47:09.187070 7477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:09.199616 7477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:09.202770 7477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:09.202932 7477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:09.203524 7477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:09.204335 7477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:09.204378 7477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:09.204416 7477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:09.204435 7477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:09.212671 7477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45527
I20260924 04:47:09.212754 7605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45527 every 8 connection(s)
I20260924 04:47:09.213653 7477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:47:09.219935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 7477
I20260924 04:47:09.220104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 6828
Sep 24 04:47:09 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 04:47:09 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:47:09 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:47:09.226933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45177
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42905
--webserver_port=43165
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42905,127.0.98.61:34359
--tserver_master_addrs=127.0.98.62:42905,127.0.98.61:34359,127.0.98.60:38771 with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:47:09.230295 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.215707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43931 (local address 127.0.98.60:38771)
0924 04:47:09.215923 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.215928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.217099 (+ 1171us) server_negotiation.cc:407] Connection header received
0924 04:47:09.217927 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.217930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.217978 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.218043 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.219348 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.220229 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.221828 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.222227 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.222307 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.226229 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.226232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.226234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.226262 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.227962 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.228570 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.228574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.228576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.228660 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.229068 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.229072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.229073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.229291 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.229356 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.230056 (+ 700us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.230169 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260924 04:47:09.230715 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.215589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40085 (local address 127.0.98.62:42905)
0924 04:47:09.215714 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.215719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.216632 (+ 913us) server_negotiation.cc:407] Connection header received
0924 04:47:09.217821 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.217825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.217882 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.217954 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.219385 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.220183 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.223694 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.223932 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.224020 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.226849 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.226852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.226855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.226886 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.228693 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.229234 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.229238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.229239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.229322 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.229636 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.229639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.229641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.229842 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.229896 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.230442 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.230562 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:47:09.230715 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.215565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57045 (local address 127.0.98.61:34359)
0924 04:47:09.215840 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.215844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.216823 (+ 979us) server_negotiation.cc:407] Connection header received
0924 04:47:09.218159 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.218163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.218209 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.218288 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.219381 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.219984 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.221594 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.221912 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.221988 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.226380 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.226384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.226387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.226415 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.228683 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.229234 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.229238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.229239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.229322 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.229638 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.229641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.229642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.230057 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.230145 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.230442 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.230562 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260924 04:47:09.231861 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.215950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38771 (local address 127.0.98.2:43931)
0924 04:47:09.217010 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.217022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.217608 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.218124 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.218129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.218612 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:09.219210 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.219224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.220809 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.220812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.221645 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.221656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.222464 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.222467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.223106 (+ 639us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.223556 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:09.223586 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:09.225811 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:09.228087 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.228094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.228113 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.228452 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.228766 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.228769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.228771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.228966 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.229432 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:09.229439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:09.229766 (+ 327us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.230522 (+ 756us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.230914 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":754,"spinlock_wait_cycles":25472,"thread_start_us":108,"threads_started":1}
I20260924 04:47:09.232100 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.215950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34359 (local address 127.0.98.2:57045)
0924 04:47:09.216753 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.216765 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.217608 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.218322 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.218325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.218612 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:09.219223 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.219229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.220158 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.220161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.221460 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.221471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.222183 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.222187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.222465 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.223556 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:09.223586 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:09.226225 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:09.228786 (+ 2561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.228790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.228792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.229122 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.229429 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.229432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.229434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.229548 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.230149 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:09.230152 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:09.230241 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.230845 (+ 604us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.231021 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":172,"spinlock_wait_cycles":43264,"thread_start_us":144,"threads_started":1}
I20260924 04:47:09.233062 7606 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38771
I20260924 04:47:09.233294 7606 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:09.233767 7606 heartbeater.cc:511] Master 127.0.98.60:38771 requested a full tablet report, sending...
I20260924 04:47:09.234218 7608 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34359
I20260924 04:47:09.234297 7608 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:09.234467 7608 heartbeater.cc:511] Master 127.0.98.61:34359 requested a full tablet report, sending...
I20260924 04:47:09.234823 7290 ts_manager.cc:194] Registered new tserver with Master: 4aa7289a60014410bbe464983b693e49 (127.0.98.2:45527)
I20260924 04:47:09.235227 7104 ts_manager.cc:194] Registered new tserver with Master: 4aa7289a60014410bbe464983b693e49 (127.0.98.2:45527)
I20260924 04:47:09.235329 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.215950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.2:40085)
0924 04:47:09.216483 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.216532 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.217619 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.218121 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.218128 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.218617 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:09.219219 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.219229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.221092 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.221095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.223581 (+ 2486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.223591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.224052 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.224055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.224132 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.224563 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:09.224578 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:09.226730 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:09.228808 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.228810 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.228812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.229122 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.229429 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.229432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.229434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.229535 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.230206 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:09.230208 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:09.230273 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.230897 (+ 624us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.235089 (+ 4192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":4127,"spinlock_wait_cycles":62976,"thread_start_us":624,"threads_started":1}
I20260924 04:47:09.236140 7607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42905
I20260924 04:47:09.236223 7607 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:09.236402 7607 heartbeater.cc:511] Master 127.0.98.62:42905 requested a full tablet report, sending...
I20260924 04:47:09.237001 6496 ts_manager.cc:194] Re-registered known tserver with Master: 4aa7289a60014410bbe464983b693e49 (127.0.98.2:45527)
I20260924 04:47:09.237756 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40085
W20260924 04:47:09.339033 7613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:09.339319 7613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:09.339362 7613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:09.343427 7613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:09.343506 7613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:09.343571 7613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:47:09.349159 7613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43165
--tserver_master_addrs=127.0.98.62:42905,127.0.98.61:34359,127.0.98.60:38771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:09.350257 7613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:09.351320 7613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:09.362190 7618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:09.362380 7619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:09.362190 7621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:09.362526 7613 server_base.cc:1034] running on GCE node
I20260924 04:47:09.363040 7613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:09.363715 7613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:09.364976 7613 hybrid_clock.cc:648] HybridClock initialized: now 1790225229364945 us; error 46 us; skew 500 ppm
Sep 24 04:47:09 dist-test-slave-hvcl krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:09.367803 7613 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:47:09.368930 7613 webserver.cc:460] Webserver started at http://127.0.98.3:43165/ using document root <none> and password file <none>
I20260924 04:47:09.369479 7613 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:09.369551 7613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:09.373634 7613 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:47:09.375841 7628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:09.377323 7613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:47:09.377429 7613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "78f3d9a08a444cc59b0acd1e563b7463"
format_stamp: "Formatted at 2026-09-24 04:47:08 on dist-test-slave-hvcl"
server_key: "84c4ab4c8ac65c56e528f0dbf6eed71e"
server_key_iv: "87fb74485a0c5e52b1e449f80c89e7c6"
server_key_version: "encryptionkey@0"
I20260924 04:47:09.377974 7613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:09.387837 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.384168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38771 (local address 127.0.0.1:32814)
0924 04:47:09.384357 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.384371 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.384477 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.384745 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.384747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.384762 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:09.384871 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.384877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.385843 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.385845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.386624 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.386634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.387467 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.387470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.387546 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.387569 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.387630 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.387702 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":56,"threads_started":1}
I20260924 04:47:09.387841 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.384240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32814 (local address 127.0.98.60:38771)
0924 04:47:09.384346 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.384351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.384440 (+ 89us) server_negotiation.cc:407] Connection header received
0924 04:47:09.384567 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.384570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.384625 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.384722 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:09.384966 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.385703 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.386725 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.387351 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.387441 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.387578 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.387644 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.387702 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:47:09.388550 7300 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 04:47:09.389002 7635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 362ebf6fc3494b56afca08f7ea649efb. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } } }
I20260924 04:47:09.389138 7635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [sys.catalog]: This master's current role is: LEARNER
I20260924 04:47:09.419840 7613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:09.423077 7613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:09.423256 7613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:09.423837 7613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:09.424685 7613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:09.424731 7613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:09.424782 7613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:09.424809 7613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:09.433485 7613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45177
I20260924 04:47:09.433548 7744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45177 every 8 connection(s)
I20260924 04:47:09.434651 7613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:47:09.435904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 7613
Sep 24 04:47:09 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:47:09 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:47:09 dist-test-slave-hvcl krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:47:09.454001 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.437212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38945 (local address 127.0.98.62:42905)
0924 04:47:09.437338 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.437342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.438421 (+ 1079us) server_negotiation.cc:407] Connection header received
0924 04:47:09.439229 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.439233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.439287 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.439351 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.442705 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.443334 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.444095 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.444424 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.444517 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.450009 (+ 5492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.450012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.450014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.450039 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.451398 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.452662 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.452666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.452667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.452749 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.453051 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.453054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.453056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.453259 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.453339 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.453682 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.453836 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:47:09.454447 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.436682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48835 (local address 127.0.98.60:38771)
0924 04:47:09.436814 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.436819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.437950 (+ 1131us) server_negotiation.cc:407] Connection header received
0924 04:47:09.440789 (+ 2839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.440792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.440852 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.440909 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.441467 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.442236 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.443829 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.444160 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.444223 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.449277 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.449281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.449284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.449322 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.450897 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.452173 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.452177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.452179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.452260 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.452657 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.452660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.452662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.452895 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.452977 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.454177 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.454291 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47,"spinlock_wait_cycles":1920}
I20260924 04:47:09.455719 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.440083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.0.98.61:34359)
0924 04:47:09.445299 (+ 5216us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.445304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.446720 (+ 1416us) server_negotiation.cc:407] Connection header received
0924 04:47:09.446760 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.446804 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.446855 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.446930 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.447755 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.448526 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.449532 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.449861 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.449951 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.451524 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.451527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.451530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.451553 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.453865 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.454332 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.454335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.454337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.454415 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.454699 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.454702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.454704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.454892 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.455024 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.455459 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.455568 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5130,"thread_start_us":76,"threads_started":1}
I20260924 04:47:09.456470 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.439159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43205 (local address 127.0.98.61:34359)
0924 04:47:09.439273 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.439278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.439671 (+ 393us) server_negotiation.cc:407] Connection header received
0924 04:47:09.439849 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.439852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.439902 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.439952 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.441179 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.441980 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.444856 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.445155 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.445253 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.451674 (+ 6421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.451677 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.451680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.451701 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.453132 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.454954 (+ 1822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.454957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.454959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.455086 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.455366 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.455370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.455372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.455583 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.455647 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.456213 (+ 566us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.456323 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48,"spinlock_wait_cycles":896}
I20260924 04:47:09.456602 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.437028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38771 (local address 127.0.98.3:48835)
0924 04:47:09.437786 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.437842 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.438897 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.441045 (+ 2148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.441047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.441214 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:09.441334 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.441341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.442369 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.442372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.443709 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.443718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.444421 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.444424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.444625 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.445792 (+ 1167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:09.445807 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:09.449114 (+ 3307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:09.451073 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.451080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.451100 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.452068 (+ 968us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.452362 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.452365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.452367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.452554 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.453044 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:09.453051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:09.453390 (+ 339us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.454661 (+ 1271us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.455113 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":63,"thread_start_us":657,"threads_started":1}
I20260924 04:47:09.456890 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.438576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.0.98.62:42905)
0924 04:47:09.439455 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.439460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.444881 (+ 5421us) server_negotiation.cc:407] Connection header received
0924 04:47:09.444921 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.444924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.444972 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.445048 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.447081 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.447876 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.448947 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.449282 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.449360 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.451994 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.451998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.452000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.452026 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.454567 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.455560 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.455564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.455566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.455645 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.455926 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.455929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.455931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.456126 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.456229 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.456611 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.456742 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":82,"threads_started":1}
I20260924 04:47:09.457459 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.439165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34359 (local address 127.0.98.3:43205)
0924 04:47:09.439588 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.439601 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.439737 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.440285 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.440287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.440502 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:09.441017 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.441034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.442158 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.442162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.444733 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.444741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.445442 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.445446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.445761 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.446195 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:09.446208 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:09.451554 (+ 5346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:09.453974 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.453977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.453978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.454767 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.455171 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.455174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.455176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.455263 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.455716 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:09.455719 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:09.456038 (+ 319us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.457086 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.457221 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":1590,"thread_start_us":91,"threads_started":1}
I20260924 04:47:09.457830 7745 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38771
I20260924 04:47:09.458053 7745 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:09.458302 7747 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34359
I20260924 04:47:09.458475 7747 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:09.458631 7745 heartbeater.cc:511] Master 127.0.98.60:38771 requested a full tablet report, sending...
I20260924 04:47:09.457573 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.437028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.3:38945)
0924 04:47:09.438333 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.438347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.439121 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.439779 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.439788 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.440290 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:09.442568 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.442575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.443445 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.443447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.443986 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.443992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.444627 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.444630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.444769 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.445508 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:09.445528 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:09.449855 (+ 4327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:09.451931 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.451934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.451936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.452552 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.452856 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:09.452858 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:09.452860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:09.452941 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:09.453410 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:09.453413 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:09.453500 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.454678 (+ 1178us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.455490 (+ 812us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"mutex_wait_us":2636,"thread_start_us":111,"threads_started":1}
I20260924 04:47:09.459048 7747 heartbeater.cc:511] Master 127.0.98.61:34359 requested a full tablet report, sending...
I20260924 04:47:09.459801 7104 ts_manager.cc:194] Registered new tserver with Master: 78f3d9a08a444cc59b0acd1e563b7463 (127.0.98.3:45177)
I20260924 04:47:09.459857 7290 ts_manager.cc:194] Registered new tserver with Master: 78f3d9a08a444cc59b0acd1e563b7463 (127.0.98.3:45177)
I20260924 04:47:09.460083 7746 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42905
I20260924 04:47:09.460150 7746 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:09.460291 7746 heartbeater.cc:511] Master 127.0.98.62:42905 requested a full tablet report, sending...
I20260924 04:47:09.461206 6498 ts_manager.cc:194] Re-registered known tserver with Master: 78f3d9a08a444cc59b0acd1e563b7463 (127.0.98.3:45177)
I20260924 04:47:09.461841 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38945
I20260924 04:47:09.462105 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.440135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.0.98.60:38771)
0924 04:47:09.444511 (+ 4376us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.444514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.445729 (+ 1215us) server_negotiation.cc:407] Connection header received
0924 04:47:09.445855 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.445858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.445904 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.445978 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.447016 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.448003 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.449204 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.449507 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.450007 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.457713 (+ 7706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.457717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.457720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.457744 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.460431 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.460815 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.460818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.460820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.460919 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.461185 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.461188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.461189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.461443 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.461564 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.461771 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.461897 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4303,"thread_start_us":75,"threads_started":1}
I20260924 04:47:09.460770 6496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:47:09.464303 6496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:47:09.466734 7763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: true } } }
I20260924 04:47:09.466871 7763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:09.470077 7763 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb: attempting to promote NON_VOTER 211a15b5a8e943198a6ad0b45931d1de to VOTER
I20260924 04:47:09.470381 7763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } }
I20260924 04:47:09.471303 7114 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 FOLLOWER]: Refusing update from remote peer 362ebf6fc3494b56afca08f7ea649efb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 04:47:09.471593 7299 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 1 LEARNER]: Refusing update from remote peer 362ebf6fc3494b56afca08f7ea649efb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 04:47:09.471787 7633 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 04:47:09.472007 7764 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 04:47:09.473222 7633 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 211a15b5a8e943198a6ad0b45931d1de (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } } }
I20260924 04:47:09.474648 7633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } } }
I20260924 04:47:09.474748 7633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:09.474855 7633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } } }
I20260924 04:47:09.474920 7633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:09.474769 7300 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 211a15b5a8e943198a6ad0b45931d1de (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } } }
I20260924 04:47:09.475107 7633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } } }
I20260924 04:47:09.475181 7633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:09.476172 7114 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 211a15b5a8e943198a6ad0b45931d1de (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } } }
I20260924 04:47:09.476671 7635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } } }
I20260924 04:47:09.476764 7635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:47:09.479696 7772 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "362ebf6fc3494b56afca08f7ea649efb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } } }
I20260924 04:47:09.479816 7772 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:47:09.483419 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.478386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43471 (local address 127.0.0.1:48450)
0924 04:47:09.478603 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.478615 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.478691 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.479439 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.479442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.479460 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:09.479545 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.479550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.480866 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.480869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.481994 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.482001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.483005 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.483009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.483089 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.483112 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.483169 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.483235 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":40,"thread_start_us":82,"threads_started":1}
I20260924 04:47:09.483610 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.478043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45177 (local address 127.0.0.1:58310)
0924 04:47:09.478154 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.478170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.478249 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.479167 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.479170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.479188 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:09.479283 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.479289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.481055 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.481058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.482200 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.482209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.483243 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.483246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.483337 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.483357 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.483424 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.483492 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":24}
I20260924 04:47:09.483664 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.478721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48450 (local address 127.0.98.1:43471)
0924 04:47:09.479023 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.479036 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.479056 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:47:09.479121 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.479125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.479296 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.479674 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:09.479810 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.480728 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.482076 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.482890 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.483016 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.483169 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.483217 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.483301 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":66,"threads_started":1}
I20260924 04:47:09.484336 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.478210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58310 (local address 127.0.98.3:45177)
0924 04:47:09.478501 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.478514 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.478538 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:47:09.478613 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.478619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.478849 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.479074 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:09.479952 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.480957 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.482428 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.483118 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.483941 (+ 823us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.484082 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.484127 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.484204 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":115,"threads_started":1}
I20260924 04:47:09.486855 7679 tablet_service.cc:1467] Processing CreateTablet for tablet e0a3b55b8ff8457586ea67db85b9e229 (DEFAULT_TABLE table=test-table [id=0a307068c42e40839358ee3f697ab800]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:09.486716 7403 tablet_service.cc:1467] Processing CreateTablet for tablet e0a3b55b8ff8457586ea67db85b9e229 (DEFAULT_TABLE table=test-table [id=0a307068c42e40839358ee3f697ab800]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:09.487854 7679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a3b55b8ff8457586ea67db85b9e229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:09.487856 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a3b55b8ff8457586ea67db85b9e229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:09.488023 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.479547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.0.98.2:45527)
0924 04:47:09.479873 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.479883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.481251 (+ 1368us) server_negotiation.cc:407] Connection header received
0924 04:47:09.481482 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.481488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.481720 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.481875 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:09.482039 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.485275 (+ 3236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.486618 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.487426 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.487534 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.487680 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.487795 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.487890 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":117,"threads_started":1}
I20260924 04:47:09.488050 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.478790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45527 (local address 127.0.0.1:55170)
0924 04:47:09.481152 (+ 2362us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.481165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.481319 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.481848 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.481851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.481869 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:09.481950 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.481955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.485390 (+ 3435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.485393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.486487 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.486498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.487548 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.487551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.487649 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.487673 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.487753 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.487844 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2304,"thread_start_us":80,"threads_started":1}
I20260924 04:47:09.490904 7540 tablet_service.cc:1467] Processing CreateTablet for tablet e0a3b55b8ff8457586ea67db85b9e229 (DEFAULT_TABLE table=test-table [id=0a307068c42e40839358ee3f697ab800]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:09.491986 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a3b55b8ff8457586ea67db85b9e229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:09.495323 7778 tablet_bootstrap.cc:492] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463: Bootstrap starting.
I20260924 04:47:09.496500 7779 tablet_bootstrap.cc:492] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af: Bootstrap starting.
I20260924 04:47:09.497639 7778 tablet_bootstrap.cc:654] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:09.498476 7778 log.cc:824] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:09.498522 7779 tablet_bootstrap.cc:654] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:09.499321 7779 log.cc:824] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:09.500600 7781 tablet_bootstrap.cc:492] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49: Bootstrap starting.
I20260924 04:47:09.500742 7778 tablet_bootstrap.cc:492] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463: No bootstrap required, opened a new log
I20260924 04:47:09.500895 7778 ts_tablet_manager.cc:1397] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:47:09.501124 7779 tablet_bootstrap.cc:492] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af: No bootstrap required, opened a new log
I20260924 04:47:09.501289 7779 ts_tablet_manager.cc:1397] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:47:09.502700 7781 tablet_bootstrap.cc:654] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:09.503539 7781 log.cc:824] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:09.504657 7778 raft_consensus.cc:348] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:09.504874 7778 raft_consensus.cc:374] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:09.504926 7778 raft_consensus.cc:729] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f3d9a08a444cc59b0acd1e563b7463, State: Initialized, Role: FOLLOWER
I20260924 04:47:09.504882 7779 raft_consensus.cc:348] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:09.505136 7779 raft_consensus.cc:374] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:09.505190 7779 raft_consensus.cc:729] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca550e78f9e4458bfe8316441d9e1af, State: Initialized, Role: FOLLOWER
I20260924 04:47:09.505406 7778 consensus_queue.cc:260] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:09.505582 7781 tablet_bootstrap.cc:492] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49: No bootstrap required, opened a new log
I20260924 04:47:09.505771 7781 ts_tablet_manager.cc:1397] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:47:09.505597 7779 consensus_queue.cc:260] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:09.506366 7778 ts_tablet_manager.cc:1428] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260924 04:47:09.506626 7746 heartbeater.cc:503] Master 127.0.98.62:42905 was elected leader, sending a full tablet report...
I20260924 04:47:09.506855 7779 ts_tablet_manager.cc:1428] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:47:09.510200 7470 heartbeater.cc:503] Master 127.0.98.62:42905 was elected leader, sending a full tablet report...
I20260924 04:47:09.510093 7781 raft_consensus.cc:348] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:09.510349 7781 raft_consensus.cc:374] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:09.510406 7781 raft_consensus.cc:729] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa7289a60014410bbe464983b693e49, State: Initialized, Role: FOLLOWER
I20260924 04:47:09.510870 7781 consensus_queue.cc:260] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:09.511840 7781 ts_tablet_manager.cc:1428] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:47:09.512923 7607 heartbeater.cc:503] Master 127.0.98.62:42905 was elected leader, sending a full tablet report...
W20260924 04:47:09.525790 7472 tablet.cc:2369] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:47:09.634127 7784 raft_consensus.cc:482] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:47:09.634337 7784 raft_consensus.cc:504] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:09.635528 7784 leader_election.cc:288] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aa7289a60014410bbe464983b693e49 (127.0.98.2:45527), 3ca550e78f9e4458bfe8316441d9e1af (127.0.98.1:43471)
I20260924 04:47:09.639626 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.635779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43471 (local address 127.0.98.3:34553)
0924 04:47:09.635928 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.635943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.636090 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.636342 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.636345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.636364 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:09.636474 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.636479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.637217 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.637221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.638503 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.638513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.639207 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.639211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.639321 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.639345 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.639416 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.639485 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":21,"spinlock_wait_cycles":9984}
I20260924 04:47:09.639652 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.635967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34553 (local address 127.0.98.1:43471)
0924 04:47:09.636077 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.636081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.636094 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:47:09.636191 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.636194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.636236 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.636319 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:09.636579 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.637095 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.638639 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.639088 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.639178 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.639233 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.639444 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.639524 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:47:09.639716 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.635872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57935 (local address 127.0.98.2:45527)
0924 04:47:09.635990 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.635994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.636011 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:47:09.636172 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.636175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.636231 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.636331 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:09.636661 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.637256 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.638663 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.639211 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.639315 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.639376 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.639546 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.639615 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:47:09.639760 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.635749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45527 (local address 127.0.98.3:57935)
0924 04:47:09.635928 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:09.635943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:09.636038 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:09.636439 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:09.636441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:09.636458 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:09.636547 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.636552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.637356 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.637359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.638503 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:09.638513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.639325 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.639328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:09.639423 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.639447 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:47:09.639518 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:47:09.639593 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":26,"spinlock_wait_cycles":14464}
I20260924 04:47:09.640349 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a3b55b8ff8457586ea67db85b9e229" candidate_uuid: "78f3d9a08a444cc59b0acd1e563b7463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ca550e78f9e4458bfe8316441d9e1af" is_pre_election: true
I20260924 04:47:09.640355 7560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a3b55b8ff8457586ea67db85b9e229" candidate_uuid: "78f3d9a08a444cc59b0acd1e563b7463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aa7289a60014410bbe464983b693e49" is_pre_election: true
I20260924 04:47:09.640626 7560 raft_consensus.cc:2454] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78f3d9a08a444cc59b0acd1e563b7463 in term 0.
I20260924 04:47:09.640626 7423 raft_consensus.cc:2454] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78f3d9a08a444cc59b0acd1e563b7463 in term 0.
I20260924 04:47:09.641038 7629 leader_election.cc:302] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ca550e78f9e4458bfe8316441d9e1af, 78f3d9a08a444cc59b0acd1e563b7463; no voters:
I20260924 04:47:09.641292 7784 raft_consensus.cc:2781] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:47:09.641376 7784 raft_consensus.cc:482] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:47:09.641422 7784 raft_consensus.cc:3037] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:09.642457 7784 raft_consensus.cc:504] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:09.642774 7784 leader_election.cc:288] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [CANDIDATE]: Term 1 election: Requested vote from peers 4aa7289a60014410bbe464983b693e49 (127.0.98.2:45527), 3ca550e78f9e4458bfe8316441d9e1af (127.0.98.1:43471)
I20260924 04:47:09.643203 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a3b55b8ff8457586ea67db85b9e229" candidate_uuid: "78f3d9a08a444cc59b0acd1e563b7463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ca550e78f9e4458bfe8316441d9e1af"
I20260924 04:47:09.643206 7560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a3b55b8ff8457586ea67db85b9e229" candidate_uuid: "78f3d9a08a444cc59b0acd1e563b7463" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aa7289a60014410bbe464983b693e49"
I20260924 04:47:09.643308 7423 raft_consensus.cc:3037] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:09.643316 7560 raft_consensus.cc:3037] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:09.644300 7423 raft_consensus.cc:2454] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78f3d9a08a444cc59b0acd1e563b7463 in term 1.
I20260924 04:47:09.644577 7629 leader_election.cc:302] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ca550e78f9e4458bfe8316441d9e1af, 78f3d9a08a444cc59b0acd1e563b7463; no voters:
I20260924 04:47:09.644593 7560 raft_consensus.cc:2454] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78f3d9a08a444cc59b0acd1e563b7463 in term 1.
I20260924 04:47:09.644733 7784 raft_consensus.cc:2781] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:09.644930 7784 raft_consensus.cc:686] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [term 1 LEADER]: Becoming Leader. State: Replica: 78f3d9a08a444cc59b0acd1e563b7463, State: Running, Role: LEADER
I20260924 04:47:09.645265 7784 consensus_queue.cc:237] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:09.648782 6496 catalog_manager.cc:5495] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78f3d9a08a444cc59b0acd1e563b7463 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "78f3d9a08a444cc59b0acd1e563b7463" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20260924 04:47:09.673759 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.670211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.0.98.3:45177)
0924 04:47:09.670335 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.670340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.670354 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:47:09.670444 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.670446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.670503 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.670591 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:47:09.670835 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.671381 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.672213 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.672423 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.672507 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.672792 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:47:09.673297 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:47:09.673422 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.673518 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.673590 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:47:09.682783 7560 raft_consensus.cc:1261] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49 [term 1 FOLLOWER]: Refusing update from remote peer 78f3d9a08a444cc59b0acd1e563b7463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:09.682993 7423 raft_consensus.cc:1261] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af [term 1 FOLLOWER]: Refusing update from remote peer 78f3d9a08a444cc59b0acd1e563b7463: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:09.683581 7784 consensus_queue.cc:1035] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:09.683763 7787 consensus_queue.cc:1035] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:47:09.690027 7748 tablet.cc:2369] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:47:09.692691 7792 mvcc.cc:205] Tried to move back new op lower bound from 7332762540772741120 to 7332762540629676032. Current Snapshot: MvccSnapshot[applied={T|T < 7332762540772741120}]
I20260924 04:47:09.697338 7791 mvcc.cc:205] Tried to move back new op lower bound from 7332762540772741120 to 7332762540629676032. Current Snapshot: MvccSnapshot[applied={T|T < 7332762540772741120}]
I20260924 04:47:09.702643 6496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:09.702801 6496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:09.706825 6496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 362ebf6fc3494b56afca08f7ea649efb: Sending DeleteTablet for 3 replicas of tablet e0a3b55b8ff8457586ea67db85b9e229
I20260924 04:47:09.707819 7679 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a3b55b8ff8457586ea67db85b9e229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58310
I20260924 04:47:09.707788 7540 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a3b55b8ff8457586ea67db85b9e229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55170
I20260924 04:47:09.708308 7403 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a3b55b8ff8457586ea67db85b9e229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48450
I20260924 04:47:09.709592 7803 tablet_replica.cc:330] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af: stopping tablet replica
I20260924 04:47:09.709774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 6466
I20260924 04:47:09.710685 7803 raft_consensus.cc:2229] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:09.711093 7803 raft_consensus.cc:2258] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:09.711468 7800 tablet_replica.cc:330] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49: stopping tablet replica
I20260924 04:47:09.711771 7800 raft_consensus.cc:2229] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:09.712054 7800 raft_consensus.cc:2258] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:09.713572 7803 ts_tablet_manager.cc:1905] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:09.716259 7803 ts_tablet_manager.cc:1918] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:09.717216 7803 log.cc:1196] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-0/wal/wals/e0a3b55b8ff8457586ea67db85b9e229
I20260924 04:47:09.717547 7803 ts_tablet_manager.cc:1939] T e0a3b55b8ff8457586ea67db85b9e229 P 3ca550e78f9e4458bfe8316441d9e1af: Deleting consensus metadata
W20260924 04:47:09.718662 7359 connection.cc:441] server connection from 127.0.0.1:48450 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48450 (request call id 1) could send its response
I20260924 04:47:09.718676 7800 ts_tablet_manager.cc:1905] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:09.718986 7801 tablet_replica.cc:330] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463: stopping tablet replica
I20260924 04:47:09.719292 7801 raft_consensus.cc:2229] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:47:09.719727 7801 raft_consensus.cc:2258] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:09.721310 7801 ts_tablet_manager.cc:1905] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:09.721585 7800 ts_tablet_manager.cc:1918] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:09.721715 7800 log.cc:1196] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/e0a3b55b8ff8457586ea67db85b9e229
I20260924 04:47:09.722061 7800 ts_tablet_manager.cc:1939] T e0a3b55b8ff8457586ea67db85b9e229 P 4aa7289a60014410bbe464983b693e49: Deleting consensus metadata
W20260924 04:47:09.723186 7495 connection.cc:441] server connection from 127.0.0.1:55170 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55170 (request call id 1) could send its response
I20260924 04:47:09.723834 7801 ts_tablet_manager.cc:1918] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:09.723943 7801 log.cc:1196] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/e0a3b55b8ff8457586ea67db85b9e229
I20260924 04:47:09.724241 7801 ts_tablet_manager.cc:1939] T e0a3b55b8ff8457586ea67db85b9e229 P 78f3d9a08a444cc59b0acd1e563b7463: Deleting consensus metadata
W20260924 04:47:09.725271 7631 connection.cc:441] server connection from 127.0.0.1:58310 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58310 (request call id 1) could send its response
I20260924 04:47:09.727335 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.709453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45236 (local address 127.0.98.61:34359)
0924 04:47:09.709552 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.709556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.717206 (+ 7650us) server_negotiation.cc:407] Connection header received
0924 04:47:09.719235 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.719239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.719294 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.719348 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.720330 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.721078 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.722167 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.722499 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.722576 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.724401 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.724417 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.724420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.724446 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.725859 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.726245 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.726248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.726250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.726320 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.726557 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.726560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.726562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.726734 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.726845 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.727088 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.727189 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:47:09.736091 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.709134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.0.98.60:38771)
0924 04:47:09.709235 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.709240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.727417 (+ 18177us) server_negotiation.cc:407] Connection header received
0924 04:47:09.727601 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.727604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.727664 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.727717 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.728552 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.729266 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.730342 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.730675 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.730745 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.732318 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.732321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.732323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.732347 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.734226 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.734592 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.734596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.734598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.734675 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.734955 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.734958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.734960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.735338 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.735459 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.735803 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.735925 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 04:47:09.737725 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.728261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.0.98.60:38771)
0924 04:47:09.728441 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.728445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.729327 (+ 882us) server_negotiation.cc:407] Connection header received
0924 04:47:09.729489 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.729492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.729532 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.729578 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.730375 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.731076 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.732300 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.732611 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.732681 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.734251 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.734254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.734257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.734278 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.736057 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.736510 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.736513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.736515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.736589 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.736864 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.736867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.736868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.737084 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.737170 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.737497 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.737588 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 04:47:09.737964 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:09.727316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45250 (local address 127.0.98.61:34359)
0924 04:47:09.727420 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:09.727424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:09.728986 (+ 1562us) server_negotiation.cc:407] Connection header received
0924 04:47:09.729138 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:09.729141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:09.729184 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:09.729228 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:09.730201 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.731021 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.732282 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:09.732583 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:09.732652 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:09.734087 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:09.734091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:09.734094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:09.734117 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:09.735871 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.736630 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.736634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.736636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.736720 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:09.736997 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:09.737000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:09.737002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:09.737184 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:09.737273 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:09.737683 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 04:47:09.737799 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:47:10.696362 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:10.695926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.1:46423)
0924 04:47:10.696159 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:10.696247 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":82,"threads_started":1}
W20260924 04:47:10.696676 7470 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42905 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
I20260924 04:47:10.698346 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:10.697921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.2:57249)
0924 04:47:10.698173 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:10.698249 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":107,"threads_started":1}
W20260924 04:47:10.698729 7607 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42905 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
I20260924 04:47:10.699153 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:10.698586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.3:47527)
0924 04:47:10.698954 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:10.699065 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":64,"threads_started":1}
W20260924 04:47:10.699365 7746 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42905 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
I20260924 04:47:11.207276 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 362ebf6fc3494b56afca08f7ea649efb)
I20260924 04:47:11.207474 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } }
I20260924 04:47:11.207665 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 362ebf6fc3494b56afca08f7ea649efb)
I20260924 04:47:11.207837 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } }
I20260924 04:47:11.208520 7814 leader_election.cc:288] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 362ebf6fc3494b56afca08f7ea649efb (127.0.98.62:42905), 211a15b5a8e943198a6ad0b45931d1de (127.0.98.60:38771)
I20260924 04:47:11.208765 7815 leader_election.cc:288] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 362ebf6fc3494b56afca08f7ea649efb (127.0.98.62:42905), aab32ac08cfb45928b8ceaeb6eba4d83 (127.0.98.61:34359)
I20260924 04:47:11.209220 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.208851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:48972)
0924 04:47:11.209041 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:11.209119 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":51,"threads_started":1}
I20260924 04:47:11.209419 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.208685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:48962)
0924 04:47:11.209269 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:11.209337 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":69,"threads_started":1}
W20260924 04:47:11.209398 7277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
W20260924 04:47:11.209564 7091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
I20260924 04:47:11.210341 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.210152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:48980)
0924 04:47:11.210231 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:11.210281 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 04:47:11.210403 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.210174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:48982)
0924 04:47:11.210298 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:11.210343 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 04:47:11.210430 7277 leader_election.cc:334] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 362ebf6fc3494b56afca08f7ea649efb (127.0.98.62:42905): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
W20260924 04:47:11.210492 7091 leader_election.cc:334] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 362ebf6fc3494b56afca08f7ea649efb (127.0.98.62:42905): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
I20260924 04:47:11.212908 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.208760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38771 (local address 127.0.0.1:32854)
0924 04:47:11.209342 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:11.209356 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:11.209457 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:11.209873 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:11.209876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:11.209895 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:11.210004 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:11.210011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:11.211015 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:11.211018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:11.211869 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:11.211878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:11.212470 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:11.212474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:11.212633 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:11.212652 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:47:11.212716 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:47:11.212785 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":39,"thread_start_us":64,"threads_started":1}
I20260924 04:47:11.213245 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.209089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.0.98.61:34359)
0924 04:47:11.209658 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:11.209662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:11.209675 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:47:11.209717 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:11.209719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:11.209761 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:11.209841 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:11.210682 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:11.211271 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:11.212234 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:11.212705 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:11.212793 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:11.212844 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:11.213059 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 04:47:11.213141 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":763,"server-negotiator.queue_time_us":512,"thread_start_us":53,"threads_started":1}
I20260924 04:47:11.213289 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.209018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34359 (local address 127.0.0.1:45260)
0924 04:47:11.209349 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:11.209361 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:11.209458 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:11.209947 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:11.209950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:11.209967 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:11.210060 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:11.210066 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:11.211368 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:11.211372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:11.212128 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:11.212136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:11.212795 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:11.212799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:11.212890 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:11.212911 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:47:11.213077 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 04:47:11.213147 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":71,"threads_started":1}
I20260924 04:47:11.213502 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.209054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.0.98.60:38771)
0924 04:47:11.209647 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:11.209652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:11.209664 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:47:11.209706 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:11.209709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:11.209754 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:11.210109 (+ 355us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:11.210218 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:11.210891 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:11.211982 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:11.212368 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:11.212457 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:11.212516 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:11.212905 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 04:47:11.213154 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":69,"threads_started":1}
I20260924 04:47:11.213686 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "211a15b5a8e943198a6ad0b45931d1de" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" is_pre_election: true
I20260924 04:47:11.213874 7114 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 211a15b5a8e943198a6ad0b45931d1de in term 1.
I20260924 04:47:11.213905 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "211a15b5a8e943198a6ad0b45931d1de" is_pre_election: true
I20260924 04:47:11.214067 7300 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aab32ac08cfb45928b8ceaeb6eba4d83 in term 1.
I20260924 04:47:11.214238 7274 leader_election.cc:302] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 211a15b5a8e943198a6ad0b45931d1de, aab32ac08cfb45928b8ceaeb6eba4d83; no voters: 362ebf6fc3494b56afca08f7ea649efb
I20260924 04:47:11.214396 7089 leader_election.cc:302] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 211a15b5a8e943198a6ad0b45931d1de, aab32ac08cfb45928b8ceaeb6eba4d83; no voters: 362ebf6fc3494b56afca08f7ea649efb
I20260924 04:47:11.214448 7815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 04:47:11.214491 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 1 FOLLOWER]: Starting leader election (detected failure of leader 362ebf6fc3494b56afca08f7ea649efb)
I20260924 04:47:11.214525 7815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:47:11.214695 7814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 04:47:11.214757 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 362ebf6fc3494b56afca08f7ea649efb)
I20260924 04:47:11.214793 7814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:47:11.215715 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } }
I20260924 04:47:11.215740 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } }
I20260924 04:47:11.216066 7815 leader_election.cc:288] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [CANDIDATE]: Term 2 election: Requested vote from peers 362ebf6fc3494b56afca08f7ea649efb (127.0.98.62:42905), aab32ac08cfb45928b8ceaeb6eba4d83 (127.0.98.61:34359)
I20260924 04:47:11.216403 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.216193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:48990)
0924 04:47:11.216291 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:11.216343 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 04:47:11.216543 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "211a15b5a8e943198a6ad0b45931d1de"
I20260924 04:47:11.216624 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.216244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:48998)
0924 04:47:11.216451 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:11.216501 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260924 04:47:11.216737 7300 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate aab32ac08cfb45928b8ceaeb6eba4d83 in current term 2: Already voted for candidate 211a15b5a8e943198a6ad0b45931d1de in this term.
I20260924 04:47:11.216951 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "211a15b5a8e943198a6ad0b45931d1de" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83"
I20260924 04:47:11.217105 7114 raft_consensus.cc:2379] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 211a15b5a8e943198a6ad0b45931d1de in current term 2: Already voted for candidate aab32ac08cfb45928b8ceaeb6eba4d83 in this term.
I20260924 04:47:11.217260 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.217019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:49004)
0924 04:47:11.217157 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:11.217203 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 04:47:11.217271 7814 leader_election.cc:288] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [CANDIDATE]: Term 2 election: Requested vote from peers 362ebf6fc3494b56afca08f7ea649efb (127.0.98.62:42905), 211a15b5a8e943198a6ad0b45931d1de (127.0.98.60:38771)
W20260924 04:47:11.217330 7277 leader_election.cc:334] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 362ebf6fc3494b56afca08f7ea649efb (127.0.98.62:42905): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
I20260924 04:47:11.217437 7274 leader_election.cc:302] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 211a15b5a8e943198a6ad0b45931d1de; no voters: 362ebf6fc3494b56afca08f7ea649efb, aab32ac08cfb45928b8ceaeb6eba4d83
I20260924 04:47:11.217608 7815 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 04:47:11.217767 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.217570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:49008)
0924 04:47:11.217663 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:11.217712 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 04:47:11.217875 7091 leader_election.cc:334] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 362ebf6fc3494b56afca08f7ea649efb (127.0.98.62:42905): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
I20260924 04:47:11.217944 7091 leader_election.cc:302] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aab32ac08cfb45928b8ceaeb6eba4d83; no voters: 211a15b5a8e943198a6ad0b45931d1de, 362ebf6fc3494b56afca08f7ea649efb
I20260924 04:47:11.218075 7814 raft_consensus.cc:2726] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 04:47:11.697791 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.697258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.1:59289)
0924 04:47:11.697562 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:11.697662 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
I20260924 04:47:11.699676 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.699272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.2:43869)
0924 04:47:11.699493 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:11.699574 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
I20260924 04:47:11.700255 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:11.699912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.3:58099)
0924 04:47:11.700134 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:11.700183 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260924 04:47:12.699016 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.698469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.1:37289)
0924 04:47:12.698797 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.698872 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":71,"threads_started":1}
I20260924 04:47:12.700702 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.700268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.2:55201)
0924 04:47:12.700541 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.700604 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":69,"threads_started":1}
I20260924 04:47:12.701480 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.701129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.3:54185)
0924 04:47:12.701337 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.701397 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260924 04:47:12.725598 7830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:47:12.725759 7830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } }
I20260924 04:47:12.726204 7830 leader_election.cc:288] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 362ebf6fc3494b56afca08f7ea649efb (127.0.98.62:42905), aab32ac08cfb45928b8ceaeb6eba4d83 (127.0.98.61:34359)
I20260924 04:47:12.726696 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "211a15b5a8e943198a6ad0b45931d1de" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" is_pre_election: true
I20260924 04:47:12.726835 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.726420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:49234)
0924 04:47:12.726656 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.726727 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":62,"threads_started":1}
I20260924 04:47:12.726853 7114 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 211a15b5a8e943198a6ad0b45931d1de in term 2.
I20260924 04:47:12.727147 7274 leader_election.cc:302] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 211a15b5a8e943198a6ad0b45931d1de, aab32ac08cfb45928b8ceaeb6eba4d83; no voters:
I20260924 04:47:12.727339 7830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 04:47:12.727391 7830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:47:12.727427 7830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:47:12.727653 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.727450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:49246)
0924 04:47:12.727544 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.727593 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 04:47:12.727757 7277 leader_election.cc:334] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 362ebf6fc3494b56afca08f7ea649efb (127.0.98.62:42905): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
I20260924 04:47:12.728542 7830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } }
I20260924 04:47:12.728847 7830 leader_election.cc:288] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [CANDIDATE]: Term 3 election: Requested vote from peers 362ebf6fc3494b56afca08f7ea649efb (127.0.98.62:42905), aab32ac08cfb45928b8ceaeb6eba4d83 (127.0.98.61:34359)
I20260924 04:47:12.729143 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.728932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:49250)
0924 04:47:12.729037 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.729083 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 04:47:12.729180 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "211a15b5a8e943198a6ad0b45931d1de" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83"
I20260924 04:47:12.729321 7114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:47:12.729698 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.729521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:49262)
0924 04:47:12.729610 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.729652 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 04:47:12.729785 7277 leader_election.cc:334] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 362ebf6fc3494b56afca08f7ea649efb (127.0.98.62:42905): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
I20260924 04:47:12.730306 7114 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 211a15b5a8e943198a6ad0b45931d1de in term 3.
I20260924 04:47:12.730571 7274 leader_election.cc:302] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 211a15b5a8e943198a6ad0b45931d1de, aab32ac08cfb45928b8ceaeb6eba4d83; no voters: 362ebf6fc3494b56afca08f7ea649efb
I20260924 04:47:12.730736 7830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:47:12.730949 7830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [term 3 LEADER]: Becoming Leader. State: Replica: 211a15b5a8e943198a6ad0b45931d1de, State: Running, Role: LEADER
I20260924 04:47:12.731218 7830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } }
I20260924 04:47:12.732579 7833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 211a15b5a8e943198a6ad0b45931d1de. Latest consensus state: current_term: 3 leader_uuid: "211a15b5a8e943198a6ad0b45931d1de" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } } }
I20260924 04:47:12.732692 7833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:12.732985 7835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:12.735268 7835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:12.735679 7835 catalog_manager.cc:1259] Loaded cluster ID: 2723da5504bc4f1a893a21ee44f47e03
I20260924 04:47:12.735721 7835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:12.736260 7835 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:12.736660 7835 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: Loaded TSK: 0
I20260924 04:47:12.737301 7835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:47:12.788160 7290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:47:12.791299 7290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:47:12.794340 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.794098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:49278)
0924 04:47:12.794207 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.794259 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 04:47:12.794382 7114 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [term 3 FOLLOWER]: Refusing update from remote peer 211a15b5a8e943198a6ad0b45931d1de: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 04:47:12.794859 7830 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 04:47:12.795004 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.794808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.0.1:49284)
0924 04:47:12.794884 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.794928 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 04:47:12.795357 7277 consensus_peers.cc:487] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de -> Peer 362ebf6fc3494b56afca08f7ea649efb (127.0.98.62:42905): Couldn't send request to peer 362ebf6fc3494b56afca08f7ea649efb. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 04:47:12.796226 7837 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 211a15b5a8e943198a6ad0b45931d1de. Latest consensus state: current_term: 3 leader_uuid: "211a15b5a8e943198a6ad0b45931d1de" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } } }
I20260924 04:47:12.796403 7837 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:47:12.797415 7830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "211a15b5a8e943198a6ad0b45931d1de" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } } }
I20260924 04:47:12.797523 7830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:12.797907 7837 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "211a15b5a8e943198a6ad0b45931d1de" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "362ebf6fc3494b56afca08f7ea649efb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42905 } } peers { permanent_uuid: "aab32ac08cfb45928b8ceaeb6eba4d83" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34359 } attrs { promote: false } } peers { permanent_uuid: "211a15b5a8e943198a6ad0b45931d1de" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38771 } attrs { promote: false } } }
I20260924 04:47:12.798022 7837 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab32ac08cfb45928b8ceaeb6eba4d83 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:47:12.808113 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.803912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45527 (local address 127.0.0.1:55178)
0924 04:47:12.804366 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.804379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:12.804458 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:12.805018 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:12.805021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:12.805038 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:12.805125 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:12.805130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.805944 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.805947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:12.806814 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:12.806825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.807675 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.807679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:12.807765 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:12.807789 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:47:12.807865 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 04:47:12.807942 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":24,"thread_start_us":93,"threads_started":1}
I20260924 04:47:12.808233 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.803744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48458 (local address 127.0.98.1:43471)
0924 04:47:12.803966 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:12.803969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:12.804303 (+ 334us) server_negotiation.cc:407] Connection header received
0924 04:47:12.804355 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:12.804358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:12.804400 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:12.804473 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:12.804784 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.805439 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.807095 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.807672 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.807776 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:12.807842 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:12.808010 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 04:47:12.808091 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":67,"threads_started":1}
I20260924 04:47:12.808274 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.803607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43471 (local address 127.0.0.1:48458)
0924 04:47:12.804132 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.804145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:12.804227 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:12.804573 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:12.804577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:12.804592 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:12.804678 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:12.804684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.805528 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.805531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:12.806932 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:12.806939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.807798 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.807802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:12.807899 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:12.807921 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:47:12.807983 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:47:12.808075 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":88,"thread_start_us":93,"threads_started":1}
I20260924 04:47:12.808856 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.803161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45177 (local address 127.0.0.1:58330)
0924 04:47:12.803264 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.803279 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:12.803390 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:12.804063 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:12.804066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:12.804086 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:12.804197 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:12.804203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.805544 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.805547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:12.806908 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:12.806920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.808515 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.808518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:12.808590 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:12.808612 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:47:12.808683 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:47:12.808748 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":88}
I20260924 04:47:12.808943 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.803585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.0.98.3:45177)
0924 04:47:12.803824 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:12.803829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:12.803843 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:47:12.803895 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:12.803898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:12.803947 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:12.804031 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:12.804702 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.805458 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.807813 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.808415 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.808515 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:12.808575 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:12.808744 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 04:47:12.808822 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
I20260924 04:47:12.808974 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.804509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55178 (local address 127.0.98.2:45527)
0924 04:47:12.804802 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:12.804806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:12.804817 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:47:12.804871 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:12.804874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:12.804917 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:12.805001 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:12.805222 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.805841 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.806958 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.807576 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.807679 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:12.807751 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:12.808764 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0924 04:47:12.808843 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":76,"threads_started":1}
I20260924 04:47:12.809517 7403 tablet_service.cc:1467] Processing CreateTablet for tablet b8deeb5343dd4f4f93485f367c1c6121 (DEFAULT_TABLE table=test-table [id=57f434f4b95146b7b20424ea6651b079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:12.809540 7540 tablet_service.cc:1467] Processing CreateTablet for tablet b8deeb5343dd4f4f93485f367c1c6121 (DEFAULT_TABLE table=test-table [id=57f434f4b95146b7b20424ea6651b079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:12.809702 7679 tablet_service.cc:1467] Processing CreateTablet for tablet b8deeb5343dd4f4f93485f367c1c6121 (DEFAULT_TABLE table=test-table [id=57f434f4b95146b7b20424ea6651b079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:12.809795 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8deeb5343dd4f4f93485f367c1c6121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:12.809800 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8deeb5343dd4f4f93485f367c1c6121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:12.809912 7679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8deeb5343dd4f4f93485f367c1c6121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:12.811882 7848 tablet_bootstrap.cc:492] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49: Bootstrap starting.
I20260924 04:47:12.812252 7850 tablet_bootstrap.cc:492] T b8deeb5343dd4f4f93485f367c1c6121 P 3ca550e78f9e4458bfe8316441d9e1af: Bootstrap starting.
I20260924 04:47:12.813046 7848 tablet_bootstrap.cc:654] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:12.813046 7850 tablet_bootstrap.cc:654] T b8deeb5343dd4f4f93485f367c1c6121 P 3ca550e78f9e4458bfe8316441d9e1af: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:12.813755 7849 tablet_bootstrap.cc:492] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463: Bootstrap starting.
I20260924 04:47:12.814312 7848 tablet_bootstrap.cc:492] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49: No bootstrap required, opened a new log
I20260924 04:47:12.814404 7848 ts_tablet_manager.cc:1397] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:47:12.814612 7849 tablet_bootstrap.cc:654] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:12.814844 7848 raft_consensus.cc:348] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:12.815003 7848 raft_consensus.cc:374] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:12.815049 7848 raft_consensus.cc:729] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa7289a60014410bbe464983b693e49, State: Initialized, Role: FOLLOWER
I20260924 04:47:12.815218 7848 consensus_queue.cc:260] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:12.815613 7850 tablet_bootstrap.cc:492] T b8deeb5343dd4f4f93485f367c1c6121 P 3ca550e78f9e4458bfe8316441d9e1af: No bootstrap required, opened a new log
I20260924 04:47:12.815618 7848 ts_tablet_manager.cc:1428] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 04:47:12.815729 7849 tablet_bootstrap.cc:492] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463: No bootstrap required, opened a new log
I20260924 04:47:12.815747 7850 ts_tablet_manager.cc:1397] T b8deeb5343dd4f4f93485f367c1c6121 P 3ca550e78f9e4458bfe8316441d9e1af: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 04:47:12.815816 7849 ts_tablet_manager.cc:1397] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:12.816210 7849 raft_consensus.cc:348] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:12.816273 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.816025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.2:45689)
0924 04:47:12.816135 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.816196 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 04:47:12.816311 7849 raft_consensus.cc:374] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:12.816342 7849 raft_consensus.cc:729] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f3d9a08a444cc59b0acd1e563b7463, State: Initialized, Role: FOLLOWER
I20260924 04:47:12.816452 7849 consensus_queue.cc:260] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:12.816788 7849 ts_tablet_manager.cc:1428] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:12.817972 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.817719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.3:37385)
0924 04:47:12.817845 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.817901 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 04:47:12.818243 7606 heartbeater.cc:503] Master 127.0.98.60:38771 was elected leader, sending a full tablet report...
I20260924 04:47:12.818627 7850 raft_consensus.cc:348] T b8deeb5343dd4f4f93485f367c1c6121 P 3ca550e78f9e4458bfe8316441d9e1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:12.818773 7850 raft_consensus.cc:374] T b8deeb5343dd4f4f93485f367c1c6121 P 3ca550e78f9e4458bfe8316441d9e1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:12.819118 7850 raft_consensus.cc:729] T b8deeb5343dd4f4f93485f367c1c6121 P 3ca550e78f9e4458bfe8316441d9e1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca550e78f9e4458bfe8316441d9e1af, State: Initialized, Role: FOLLOWER
I20260924 04:47:12.819186 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.818947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.2:35375)
0924 04:47:12.819076 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.819125 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 04:47:12.819710 7850 consensus_queue.cc:260] T b8deeb5343dd4f4f93485f367c1c6121 P 3ca550e78f9e4458bfe8316441d9e1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:12.820034 7850 ts_tablet_manager.cc:1428] T b8deeb5343dd4f4f93485f367c1c6121 P 3ca550e78f9e4458bfe8316441d9e1af: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260924 04:47:12.820708 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.820464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.1:57579)
0924 04:47:12.820594 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.820643 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 04:47:12.821285 7469 heartbeater.cc:503] Master 127.0.98.60:38771 was elected leader, sending a full tablet report...
I20260924 04:47:12.821566 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.821317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.1:60973)
0924 04:47:12.821456 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.821504 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 04:47:12.958854 7854 raft_consensus.cc:482] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:47:12.959010 7854 raft_consensus.cc:504] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:12.960062 7854 leader_election.cc:288] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78f3d9a08a444cc59b0acd1e563b7463 (127.0.98.3:45177), 3ca550e78f9e4458bfe8316441d9e1af (127.0.98.1:43471)
I20260924 04:47:12.964668 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.960449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42841 (local address 127.0.98.3:45177)
0924 04:47:12.960569 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:12.960573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:12.960591 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:47:12.960643 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:12.960645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:12.960686 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:12.960771 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:12.961051 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.961744 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.963432 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.964035 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.964141 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:12.964207 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:12.964481 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 04:47:12.964549 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:47:12.964668 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.960441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53997 (local address 127.0.98.1:43471)
0924 04:47:12.960569 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:12.960573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:12.960920 (+ 347us) server_negotiation.cc:407] Connection header received
0924 04:47:12.961046 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:12.961049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:12.961107 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:12.961181 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:12.961451 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.962048 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.963432 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.964035 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.964141 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:12.964211 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:12.964464 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 04:47:12.964540 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:47:12.964671 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.960346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45177 (local address 127.0.98.2:42841)
0924 04:47:12.960440 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.960454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:12.960550 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:12.960797 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:12.960800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:12.960820 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:12.960946 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:12.960952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.961915 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.961917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:12.963304 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:12.963317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.964240 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.964245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:12.964380 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:12.964398 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:47:12.964459 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 04:47:12.964513 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":58}
I20260924 04:47:12.964721 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.960343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43471 (local address 127.0.98.2:53997)
0924 04:47:12.960820 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.960833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:12.960921 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:12.961236 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:12.961238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:12.961257 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:12.961341 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:12.961346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.962160 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.962163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:12.963325 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:12.963331 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.964188 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:12.964192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:12.964303 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:12.964354 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 04:47:12.964439 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 04:47:12.964513 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":128,"thread_start_us":123,"threads_started":1}
I20260924 04:47:12.965215 7699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8deeb5343dd4f4f93485f367c1c6121" candidate_uuid: "4aa7289a60014410bbe464983b693e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f3d9a08a444cc59b0acd1e563b7463" is_pre_election: true
I20260924 04:47:12.965263 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8deeb5343dd4f4f93485f367c1c6121" candidate_uuid: "4aa7289a60014410bbe464983b693e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ca550e78f9e4458bfe8316441d9e1af" is_pre_election: true
I20260924 04:47:12.965397 7423 raft_consensus.cc:2454] T b8deeb5343dd4f4f93485f367c1c6121 P 3ca550e78f9e4458bfe8316441d9e1af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa7289a60014410bbe464983b693e49 in term 0.
I20260924 04:47:12.965456 7699 raft_consensus.cc:2454] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa7289a60014410bbe464983b693e49 in term 0.
I20260924 04:47:12.965814 7493 leader_election.cc:302] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ca550e78f9e4458bfe8316441d9e1af, 4aa7289a60014410bbe464983b693e49; no voters:
I20260924 04:47:12.966348 7854 raft_consensus.cc:2781] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:47:12.966418 7854 raft_consensus.cc:482] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:47:12.966454 7854 raft_consensus.cc:3037] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:12.967471 7854 raft_consensus.cc:504] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:12.967796 7854 leader_election.cc:288] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [CANDIDATE]: Term 1 election: Requested vote from peers 78f3d9a08a444cc59b0acd1e563b7463 (127.0.98.3:45177), 3ca550e78f9e4458bfe8316441d9e1af (127.0.98.1:43471)
W20260924 04:47:12.967883 7609 tablet.cc:2369] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:47:12.968155 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8deeb5343dd4f4f93485f367c1c6121" candidate_uuid: "4aa7289a60014410bbe464983b693e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ca550e78f9e4458bfe8316441d9e1af"
I20260924 04:47:12.968116 7699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8deeb5343dd4f4f93485f367c1c6121" candidate_uuid: "4aa7289a60014410bbe464983b693e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f3d9a08a444cc59b0acd1e563b7463"
I20260924 04:47:12.968267 7423 raft_consensus.cc:3037] T b8deeb5343dd4f4f93485f367c1c6121 P 3ca550e78f9e4458bfe8316441d9e1af [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:12.968268 7699 raft_consensus.cc:3037] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:12.969360 7699 raft_consensus.cc:2454] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa7289a60014410bbe464983b693e49 in term 1.
I20260924 04:47:12.969360 7423 raft_consensus.cc:2454] T b8deeb5343dd4f4f93485f367c1c6121 P 3ca550e78f9e4458bfe8316441d9e1af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa7289a60014410bbe464983b693e49 in term 1.
I20260924 04:47:12.969660 7493 leader_election.cc:302] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ca550e78f9e4458bfe8316441d9e1af, 4aa7289a60014410bbe464983b693e49; no voters:
I20260924 04:47:12.969839 7854 raft_consensus.cc:2781] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:12.970037 7854 raft_consensus.cc:686] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [term 1 LEADER]: Becoming Leader. State: Replica: 4aa7289a60014410bbe464983b693e49, State: Running, Role: LEADER
I20260924 04:47:12.970400 7854 consensus_queue.cc:237] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } }
I20260924 04:47:12.970764 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.970537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.2:35571)
0924 04:47:12.970634 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:12.970692 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 04:47:12.972940 7288 catalog_manager.cc:5495] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aa7289a60014410bbe464983b693e49 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4aa7289a60014410bbe464983b693e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa7289a60014410bbe464983b693e49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20260924 04:47:13.002022 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:12.998723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.0.98.2:45527)
0924 04:47:12.998854 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:12.998858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:12.998873 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:47:12.998954 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:12.998956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:12.999042 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:12.999145 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:47:12.999381 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:12.999875 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:13.000669 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:13.000883 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:13.000967 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:13.001209 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:47:13.001654 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:47:13.001754 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:13.001830 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 04:47:13.001887 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:47:13.004971 7423 raft_consensus.cc:1261] T b8deeb5343dd4f4f93485f367c1c6121 P 3ca550e78f9e4458bfe8316441d9e1af [term 1 FOLLOWER]: Refusing update from remote peer 4aa7289a60014410bbe464983b693e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:13.005198 7699 raft_consensus.cc:1261] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463 [term 1 FOLLOWER]: Refusing update from remote peer 4aa7289a60014410bbe464983b693e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:13.005569 7858 consensus_queue.cc:1035] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ca550e78f9e4458bfe8316441d9e1af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:13.005779 7854 consensus_queue.cc:1035] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f3d9a08a444cc59b0acd1e563b7463" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:13.007253 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:13.006935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.1:46887)
0924 04:47:13.007110 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:13.007173 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260924 04:47:13.007620 7745 heartbeater.cc:503] Master 127.0.98.60:38771 was elected leader, sending a full tablet report...
I20260924 04:47:13.009120 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:13.008067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.3:33189)
0924 04:47:13.008994 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:13.009050 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":859}
I20260924 04:47:13.011723 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:13.011456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.2:51271)
0924 04:47:13.011595 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:13.011650 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 04:47:13.012367 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:13.012123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.3:59919)
0924 04:47:13.012253 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:13.012301 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 04:47:13.012436 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:13.012142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42905 (local address 127.0.98.1:43339)
0924 04:47:13.012321 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:13.012369 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260924 04:47:13.014134 7863 mvcc.cc:205] Tried to move back new op lower bound from 7332762554383396864 to 7332762554249527296. Current Snapshot: MvccSnapshot[applied={T|T < 7332762554383396864}]
I20260924 04:47:13.019430 7288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:13.019574 7288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:13.031246 7288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 211a15b5a8e943198a6ad0b45931d1de: Sending DeleteTablet for 3 replicas of tablet b8deeb5343dd4f4f93485f367c1c6121
I20260924 04:47:13.032032 7679 tablet_service.cc:1514] Processing DeleteTablet for tablet b8deeb5343dd4f4f93485f367c1c6121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:13 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:58330
I20260924 04:47:13.032191 7540 tablet_service.cc:1514] Processing DeleteTablet for tablet b8deeb5343dd4f4f93485f367c1c6121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:13 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:55178
I20260924 04:47:13.032577 7872 tablet_replica.cc:330] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49: stopping tablet replica
I20260924 04:47:13.032559 7403 tablet_service.cc:1514] Processing DeleteTablet for tablet b8deeb5343dd4f4f93485f367c1c6121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:13 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48458
I20260924 04:47:13.032691 7872 raft_consensus.cc:2229] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:47:13.032994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 7249
I20260924 04:47:13.033026 7872 raft_consensus.cc:2258] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:13.033957 7871 tablet_replica.cc:330] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463: stopping tablet replica
I20260924 04:47:13.034075 7871 raft_consensus.cc:2229] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:13.034200 7871 raft_consensus.cc:2258] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:13.034374 7872 ts_tablet_manager.cc:1905] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:13.037029 7872 ts_tablet_manager.cc:1918] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:13.037143 7872 log.cc:1196] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/b8deeb5343dd4f4f93485f367c1c6121
I20260924 04:47:13.037441 7872 ts_tablet_manager.cc:1939] T b8deeb5343dd4f4f93485f367c1c6121 P 4aa7289a60014410bbe464983b693e49: Deleting consensus metadata
I20260924 04:47:13.038246 7275 catalog_manager.cc:4841] TS 4aa7289a60014410bbe464983b693e49 (127.0.98.2:45527): tablet b8deeb5343dd4f4f93485f367c1c6121 (table test-table [id=57f434f4b95146b7b20424ea6651b079]) successfully deleted
I20260924 04:47:13.038681 7871 ts_tablet_manager.cc:1905] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:13.039942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 7477
I20260924 04:47:13.040161 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:13.039862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43471 (local address 127.0.0.1:48474)
0924 04:47:13.040029 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:13.040081 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260924 04:47:13.040956 7274 catalog_manager.cc:4584] TS 3ca550e78f9e4458bfe8316441d9e1af (127.0.98.1:43471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b8deeb5343dd4f4f93485f367c1c6121: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43471: connect: Connection refused (error 111)
I20260924 04:47:13.041683 7871 ts_tablet_manager.cc:1918] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:13.041754 7871 log.cc:1196] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/b8deeb5343dd4f4f93485f367c1c6121
I20260924 04:47:13.041982 7871 ts_tablet_manager.cc:1939] T b8deeb5343dd4f4f93485f367c1c6121 P 78f3d9a08a444cc59b0acd1e563b7463: Deleting consensus metadata
I20260924 04:47:13.042833 7277 catalog_manager.cc:4841] TS 78f3d9a08a444cc59b0acd1e563b7463 (127.0.98.3:45177): tablet b8deeb5343dd4f4f93485f367c1c6121 (table test-table [id=57f434f4b95146b7b20424ea6651b079]) successfully deleted
I20260924 04:47:13.045248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 7613
I20260924 04:47:13.050835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 6962
I20260924 04:47:13.055678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 7068
2026-09-24T04:47:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6884 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:47:13 dist-test-slave-hvcl krb5kdc[7876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:47:13 dist-test-slave-hvcl krb5kdc[7876](info): set up 2 sockets
Sep 24 04:47:13 dist-test-slave-hvcl krb5kdc[7876](info): commencing operation
krb5kdc: starting...
W20260924 04:47:14.019256 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:47:14 dist-test-slave-hvcl krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:47:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 04:47:14.138367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43703
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:14.246555 7889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:14.246832 7889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:14.246871 7889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:14.250948 7889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:14.251031 7889 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:47:14.251047 7889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:14.251065 7889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:14.251080 7889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:14.251092 7889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:47:14.256361 7889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43703
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:14.257366 7889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:14.258338 7889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:14.266475 7894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:14.266517 7897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:14.266554 7895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:14.266659 7889 server_base.cc:1034] running on GCE node
I20260924 04:47:14.267344 7889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:14.268465 7889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:14.269629 7889 hybrid_clock.cc:648] HybridClock initialized: now 1790225234269614 us; error 35 us; skew 500 ppm
Sep 24 04:47:14 dist-test-slave-hvcl krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225234, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:14.272290 7889 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 04:47:14.273169 7889 webserver.cc:460] Webserver started at http://127.0.98.62:35887/ using document root <none> and password file <none>
I20260924 04:47:14.273710 7889 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:14.273772 7889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:14.273955 7889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:14.275947 7889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "4a29aecc5b5f49888ea7d3858d2b12a6"
format_stamp: "Formatted at 2026-09-24 04:47:14 on dist-test-slave-hvcl"
server_key: "f282abff7eba23d2bae2dde5bef0cc75"
server_key_iv: "af5d745b414e9116b07c5eeed8d3cdad"
server_key_version: "encryptionkey@0"
I20260924 04:47:14.276461 7889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a29aecc5b5f49888ea7d3858d2b12a6"
format_stamp: "Formatted at 2026-09-24 04:47:14 on dist-test-slave-hvcl"
server_key: "f282abff7eba23d2bae2dde5bef0cc75"
server_key_iv: "af5d745b414e9116b07c5eeed8d3cdad"
server_key_version: "encryptionkey@0"
I20260924 04:47:14.279947 7889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:47:14.282193 7904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:14.283470 7889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:14.283576 7889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "4a29aecc5b5f49888ea7d3858d2b12a6"
format_stamp: "Formatted at 2026-09-24 04:47:14 on dist-test-slave-hvcl"
server_key: "f282abff7eba23d2bae2dde5bef0cc75"
server_key_iv: "af5d745b414e9116b07c5eeed8d3cdad"
server_key_version: "encryptionkey@0"
I20260924 04:47:14.283669 7889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:14.310001 7889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:14.313416 7889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:14.313587 7889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:14.320860 7889 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43703
I20260924 04:47:14.320888 7966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43703 every 8 connection(s)
I20260924 04:47:14.321758 7889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:47:14.324327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 7889
I20260924 04:47:14.324451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:47:14.324579 7967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:14.324774 392 external_mini_cluster.cc:1428] Setting key d8a881d5549009f890c8f7cf94dae65f
I20260924 04:47:14.330813 7967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6: Bootstrap starting.
I20260924 04:47:14.333135 7967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6: Neither blocks nor log segments found. Creating new log.
Sep 24 04:47:14 dist-test-slave-hvcl krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:47:14.333956 7967 log.cc:824] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:14.336169 7967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6: No bootstrap required, opened a new log
I20260924 04:47:14.339332 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:14.326162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.0.98.62:43703)
0924 04:47:14.326582 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:14.326596 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:14.326629 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:47:14.327347 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:14.327367 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:14.327696 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:14.328005 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:14.328849 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.329819 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.330628 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.331158 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.331306 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:14.333812 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:14.333817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:14.333830 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:14.333872 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:14.336227 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:14.336656 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:14.336661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:14.336666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:14.336808 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:14.337105 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:14.337109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:14.337111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:14.337606 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:14.337782 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:14.338056 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 04:47:14.338348 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":137,"threads_started":1}
I20260924 04:47:14.339928 7967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a29aecc5b5f49888ea7d3858d2b12a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260924 04:47:14.340198 7967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:14.340301 7967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a29aecc5b5f49888ea7d3858d2b12a6, State: Initialized, Role: FOLLOWER
I20260924 04:47:14.340765 7967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a29aecc5b5f49888ea7d3858d2b12a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260924 04:47:14.340883 7967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:47:14.340939 7967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:47:14.341037 7967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:14.342016 7967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a29aecc5b5f49888ea7d3858d2b12a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260924 04:47:14.342329 7967 leader_election.cc:302] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a29aecc5b5f49888ea7d3858d2b12a6; no voters:
I20260924 04:47:14.342598 7967 leader_election.cc:288] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:47:14.342738 7972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:14.342929 7972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [term 1 LEADER]: Becoming Leader. State: Replica: 4a29aecc5b5f49888ea7d3858d2b12a6, State: Running, Role: LEADER
I20260924 04:47:14.343259 7972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a29aecc5b5f49888ea7d3858d2b12a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } }
I20260924 04:47:14.343755 7967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:14.344939 7973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a29aecc5b5f49888ea7d3858d2b12a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a29aecc5b5f49888ea7d3858d2b12a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } } }
I20260924 04:47:14.344931 7974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a29aecc5b5f49888ea7d3858d2b12a6. Latest consensus state: current_term: 1 leader_uuid: "4a29aecc5b5f49888ea7d3858d2b12a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a29aecc5b5f49888ea7d3858d2b12a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43703 } } }
I20260924 04:47:14.345072 7973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:14.345072 7974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:14.345515 7982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:14.348235 7982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:14.353070 7982 catalog_manager.cc:1347] Generated new cluster ID: 69eee90463f74bd28f74a1c94930fb4e
I20260924 04:47:14.353145 7982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:14.360991 7982 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:47:14.362057 7982 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:14.370740 7982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6: Generated new TSK 0
I20260924 04:47:14.371479 7982 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 04:47:14.403586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43703
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:14.514871 7993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:14.515165 7993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:14.515226 7993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:14.519223 7993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:14.519279 7993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:14.519333 7993 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:47:14.519395 7993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:47:14.524483 7993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:14.525512 7993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:14.526525 7993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:14.535794 7998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:14.535727 8001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:14.535951 7993 server_base.cc:1034] running on GCE node
W20260924 04:47:14.535765 7999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:14.536540 7993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:14.537166 7993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:14.538332 7993 hybrid_clock.cc:648] HybridClock initialized: now 1790225234538304 us; error 40 us; skew 500 ppm
Sep 24 04:47:14 dist-test-slave-hvcl krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225234, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:14.541077 7993 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:47:14.541983 7993 webserver.cc:460] Webserver started at http://127.0.98.1:33731/ using document root <none> and password file <none>
I20260924 04:47:14.542572 7993 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:14.542651 7993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:14.542848 7993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:14.545024 7993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "7e9c3ee360cd4a28ac4715a826332653"
format_stamp: "Formatted at 2026-09-24 04:47:14 on dist-test-slave-hvcl"
server_key: "bf25da26c5a96f33916bf2d87160a461"
server_key_iv: "384fcf3927e21b1c8c344f6f5be5182d"
server_key_version: "encryptionkey@0"
I20260924 04:47:14.545598 7993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7e9c3ee360cd4a28ac4715a826332653"
format_stamp: "Formatted at 2026-09-24 04:47:14 on dist-test-slave-hvcl"
server_key: "bf25da26c5a96f33916bf2d87160a461"
server_key_iv: "384fcf3927e21b1c8c344f6f5be5182d"
server_key_version: "encryptionkey@0"
I20260924 04:47:14.549315 7993 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:47:14.551823 8008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:14.553158 7993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:14.553263 7993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "7e9c3ee360cd4a28ac4715a826332653"
format_stamp: "Formatted at 2026-09-24 04:47:14 on dist-test-slave-hvcl"
server_key: "bf25da26c5a96f33916bf2d87160a461"
server_key_iv: "384fcf3927e21b1c8c344f6f5be5182d"
server_key_version: "encryptionkey@0"
I20260924 04:47:14.553376 7993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:14.564682 7993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:14.568044 7993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:14.568210 7993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:14.569633 7993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:14.569682 7993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:14.569723 7993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:14.569751 7993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:14.579133 7993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33507
I20260924 04:47:14.579197 8123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33507 every 8 connection(s)
Sep 24 04:47:14 dist-test-slave-hvcl krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225234, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:47:14.580212 7993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:47:14.583927 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:14.570571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47103 (local address 127.0.98.62:43703)
0924 04:47:14.570678 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:14.570682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:14.571649 (+ 967us) server_negotiation.cc:407] Connection header received
0924 04:47:14.572718 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:14.572722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:14.572781 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:14.572874 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:14.574379 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.575245 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.576717 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.577078 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.577213 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:14.580173 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:14.580177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:14.580179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:14.580205 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:14.581909 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:14.582392 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:14.582396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:14.582398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:14.582552 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:14.582865 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:14.582868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:14.582870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:14.583094 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:14.583180 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:14.583651 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 04:47:14.583772 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:47:14.584710 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:14.571016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:47103)
0924 04:47:14.571507 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:14.571547 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:14.572475 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:14.573106 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:14.573115 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:14.573529 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:14.574199 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:14.574220 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.575379 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.575383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:14.576592 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:14.576601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.577294 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.577299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:14.577501 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:14.578140 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:14.578159 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:14.579972 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:14.582032 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:14.582037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:14.582048 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:14.582283 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:14.582642 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:14.582645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:14.582647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:14.582777 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:14.583188 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:14.583195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:14.583432 (+ 237us) client_negotiation.cc:769] Sending connection context
0924 04:47:14.583596 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 04:47:14.583908 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":152,"threads_started":1}
I20260924 04:47:14.586102 8124 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43703
I20260924 04:47:14.586233 8124 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:14.586738 8124 heartbeater.cc:511] Master 127.0.98.62:43703 requested a full tablet report, sending...
I20260924 04:47:14.588207 7921 ts_manager.cc:194] Registered new tserver with Master: 7e9c3ee360cd4a28ac4715a826332653 (127.0.98.1:33507)
I20260924 04:47:14.589564 7921 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47103
I20260924 04:47:14.589929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 7993
I20260924 04:47:14.590037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:47:14.590334 392 external_mini_cluster.cc:1428] Setting key 950ff00cef834519bb41d8f25b4a8e4b
I20260924 04:47:14.595536 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:14.588134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.1:56093)
0924 04:47:14.588485 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:14.588496 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:14.588573 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:14.588749 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:14.588751 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:14.588907 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:14.589001 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:14.589006 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.589934 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.589937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:14.590729 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:14.590739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.591224 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.591228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:14.591317 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:14.591852 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:14.591866 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:14.592279 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:14.594061 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:14.594063 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:14.594065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:14.594239 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:14.594508 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:14.594510 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:14.594511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:14.594582 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:14.594947 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:14.594950 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:14.595060 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 04:47:14.595187 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 04:47:14.595320 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":88,"threads_started":1}
I20260924 04:47:14.595957 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:14.588224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56093 (local address 127.0.98.62:43703)
0924 04:47:14.588335 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:14.588351 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:14.588563 (+ 212us) server_negotiation.cc:407] Connection header received
0924 04:47:14.588631 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:14.588633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:14.588672 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:14.588744 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:14.589091 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.589799 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.590838 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.591120 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.591201 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:14.592419 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:14.592421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:14.592423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:14.592444 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:14.593955 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:14.594330 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:14.594333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:14.594335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:14.594421 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:14.594653 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:14.594655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:14.594657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:14.594845 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:14.594929 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:14.595278 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 04:47:14.595454 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 04:47:14.618037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43703
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:14.726609 8133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:14.726868 8133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:14.726910 8133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:14.730922 8133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:14.731019 8133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:14.731073 8133 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:47:14.731128 8133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:47:14.736485 8133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:14.737479 8133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:14.738440 8133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:14.747987 8138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:14.747965 8133 server_base.cc:1034] running on GCE node
W20260924 04:47:14.748001 8139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:14.747954 8141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:14.748672 8133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:14.749301 8133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:14.750455 8133 hybrid_clock.cc:648] HybridClock initialized: now 1790225234750422 us; error 44 us; skew 500 ppm
Sep 24 04:47:14 dist-test-slave-hvcl krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225234, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:14.753043 8133 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:47:14.753876 8133 webserver.cc:460] Webserver started at http://127.0.98.2:43003/ using document root <none> and password file <none>
I20260924 04:47:14.754407 8133 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:14.754469 8133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:14.754648 8133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:14.756716 8133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "6dfbe556b6874ef9ad97379390ccc222"
format_stamp: "Formatted at 2026-09-24 04:47:14 on dist-test-slave-hvcl"
server_key: "651e0acb204b0bce8487f53eee3c1d41"
server_key_iv: "4be83fced3280eefd4537910523dccea"
server_key_version: "encryptionkey@0"
I20260924 04:47:14.757243 8133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6dfbe556b6874ef9ad97379390ccc222"
format_stamp: "Formatted at 2026-09-24 04:47:14 on dist-test-slave-hvcl"
server_key: "651e0acb204b0bce8487f53eee3c1d41"
server_key_iv: "4be83fced3280eefd4537910523dccea"
server_key_version: "encryptionkey@0"
I20260924 04:47:14.760933 8133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:47:14.763331 8148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:14.764537 8133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:14.764652 8133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "6dfbe556b6874ef9ad97379390ccc222"
format_stamp: "Formatted at 2026-09-24 04:47:14 on dist-test-slave-hvcl"
server_key: "651e0acb204b0bce8487f53eee3c1d41"
server_key_iv: "4be83fced3280eefd4537910523dccea"
server_key_version: "encryptionkey@0"
I20260924 04:47:14.764753 8133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:14.788189 8133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:14.791674 8133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:14.791837 8133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:14.793201 8133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:14.793253 8133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:14.793308 8133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:14.793336 8133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:47:14 dist-test-slave-hvcl krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225234, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:47:14.803133 8133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42155
I20260924 04:47:14.803798 8263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42155 every 8 connection(s)
I20260924 04:47:14.804159 8133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:47:14.807412 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:14.794032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59817 (local address 127.0.98.62:43703)
0924 04:47:14.794152 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:14.794155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:14.795208 (+ 1053us) server_negotiation.cc:407] Connection header received
0924 04:47:14.796327 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:14.796331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:14.796386 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:14.796472 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:14.798040 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.798844 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.800544 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.800756 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.800848 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:14.803727 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:14.803731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:14.803733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:14.803759 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:14.805390 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:14.805933 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:14.805936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:14.805937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:14.806022 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:14.806352 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:14.806355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:14.806356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:14.806541 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:14.806615 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:14.807186 (+ 571us) server_negotiation.cc:299] Negotiation successful
0924 04:47:14.807292 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:47:14.808233 8157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:14.794551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:59817)
0924 04:47:14.795074 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:14.795116 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:14.796113 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:14.796692 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:14.796702 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:14.797174 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:14.797854 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:14.797877 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.799005 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.799009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:14.800428 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:14.800438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.800911 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.800916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:14.801130 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:14.801840 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:14.801859 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:14.803532 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:14.805509 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:14.805514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:14.805525 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:14.805787 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:14.806124 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:14.806127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:14.806128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:14.806263 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:14.806662 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:14.806668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:14.806910 (+ 242us) client_negotiation.cc:769] Sending connection context
0924 04:47:14.807107 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 04:47:14.807381 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":131,"threads_started":1}
I20260924 04:47:14.809384 8264 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43703
I20260924 04:47:14.809528 8264 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:14.810097 8264 heartbeater.cc:511] Master 127.0.98.62:43703 requested a full tablet report, sending...
I20260924 04:47:14.811218 7920 ts_manager.cc:194] Registered new tserver with Master: 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155)
I20260924 04:47:14.811931 7920 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59817
I20260924 04:47:14.814257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 8133
I20260924 04:47:14.814363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:47:14.814615 392 external_mini_cluster.cc:1428] Setting key 4f3420e10a6121e4aeaddf14c416376b
I20260924 04:47:14.818766 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:14.811725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.2:48491)
0924 04:47:14.812039 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:14.812054 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:14.812146 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:14.812364 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:14.812367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:14.812606 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:14.812729 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:14.812735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.813450 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.813453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:14.814003 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:14.814012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.814548 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.814551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:14.814638 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:14.815200 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:14.815217 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:14.815694 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:14.817375 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:14.817377 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:14.817379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:14.817563 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:14.817838 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:14.817840 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:14.817841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:14.817913 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:14.818346 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:14.818348 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:14.818427 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 04:47:14.818505 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:47:14.818625 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":4,"thread_start_us":96,"threads_started":1}
I20260924 04:47:14.818830 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:14.811816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48491 (local address 127.0.98.62:43703)
0924 04:47:14.811927 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:14.811931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:14.812115 (+ 184us) server_negotiation.cc:407] Connection header received
0924 04:47:14.812223 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:14.812226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:14.812272 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:14.812343 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:14.812834 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.813330 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.814130 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:14.814351 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:14.814447 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:14.815820 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:14.815824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:14.815826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:14.815849 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:14.817270 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:14.817665 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:14.817669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:14.817671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:14.817759 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:14.817992 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:14.817995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:14.817997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:14.818253 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:14.818345 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:14.818577 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 04:47:14.818685 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:47:14.842478 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43703
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:14.950934 8273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:14.951215 8273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:14.951257 8273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:14.955296 8273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:14.955360 8273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:14.955401 8273 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:47:14.955446 8273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:47:14.960770 8273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:14.961836 8273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:14.962852 8273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:47:14.972419 8273 server_base.cc:1034] running on GCE node
W20260924 04:47:14.972472 8278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:14.972361 8281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:14.972379 8279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:14.972961 8273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:14.973599 8273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:14.974779 8273 hybrid_clock.cc:648] HybridClock initialized: now 1790225234974758 us; error 34 us; skew 500 ppm
Sep 24 04:47:14 dist-test-slave-hvcl krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225234, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:14.977975 8273 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:47:14.978920 8273 webserver.cc:460] Webserver started at http://127.0.98.3:44185/ using document root <none> and password file <none>
I20260924 04:47:14.979509 8273 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:14.979579 8273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:14.979772 8273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:14.981844 8273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "ada18d69250940839ecbf049c4181dca"
format_stamp: "Formatted at 2026-09-24 04:47:14 on dist-test-slave-hvcl"
server_key: "107a3259eb37102966a9693b82c79074"
server_key_iv: "bb217e760968171d91273e6a9e55b1db"
server_key_version: "encryptionkey@0"
I20260924 04:47:14.982395 8273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ada18d69250940839ecbf049c4181dca"
format_stamp: "Formatted at 2026-09-24 04:47:14 on dist-test-slave-hvcl"
server_key: "107a3259eb37102966a9693b82c79074"
server_key_iv: "bb217e760968171d91273e6a9e55b1db"
server_key_version: "encryptionkey@0"
I20260924 04:47:14.986224 8273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:47:14.988885 8288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:14.990218 8273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:47:14.990320 8273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "ada18d69250940839ecbf049c4181dca"
format_stamp: "Formatted at 2026-09-24 04:47:14 on dist-test-slave-hvcl"
server_key: "107a3259eb37102966a9693b82c79074"
server_key_iv: "bb217e760968171d91273e6a9e55b1db"
server_key_version: "encryptionkey@0"
I20260924 04:47:14.990427 8273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:15.008524 8273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:15.011972 8273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:15.012151 8273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:15.013652 8273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:15.013702 8273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:15.013743 8273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:15.013768 8273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:47:15 dist-test-slave-hvcl krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225234, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:47:15.023656 8273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43111
I20260924 04:47:15.023967 8403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43111 every 8 connection(s)
I20260924 04:47:15.024663 8273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:47:15.026734 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.014566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54099 (local address 127.0.98.62:43703)
0924 04:47:15.014714 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.014719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.015614 (+ 895us) server_negotiation.cc:407] Connection header received
0924 04:47:15.016830 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.016834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.016909 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.017016 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:15.018624 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.019335 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.020568 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.020793 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.020881 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.023343 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:15.023346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:15.023348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:15.023377 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:15.024844 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:15.025396 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:15.025399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:15.025401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:15.025476 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:15.025800 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:15.025802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:15.025803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:15.025966 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:15.026037 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.026469 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.026616 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:47:15.027570 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.014912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:54099)
0924 04:47:15.015433 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:15.015493 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:15.016573 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:15.017228 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:15.017237 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:15.017705 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:15.018439 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.018460 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.019530 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.019533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.020445 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.020455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.020943 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.020947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.021097 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.021743 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:15.021759 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:15.023170 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:15.024953 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:15.024958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:15.024972 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:15.025283 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:15.025573 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:15.025576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:15.025578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:15.025726 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:15.026055 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:15.026060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:15.026281 (+ 221us) client_negotiation.cc:769] Sending connection context
0924 04:47:15.026436 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 04:47:15.026756 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":106,"threads_started":1}
I20260924 04:47:15.028723 8404 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43703
I20260924 04:47:15.028847 8404 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:15.029114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 8273
I20260924 04:47:15.029245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:47:15.029382 8404 heartbeater.cc:511] Master 127.0.98.62:43703 requested a full tablet report, sending...
I20260924 04:47:15.029582 392 external_mini_cluster.cc:1428] Setting key 3a501873c11d3a034c834311a8edba5e
I20260924 04:47:15.030503 7920 ts_manager.cc:194] Registered new tserver with Master: ada18d69250940839ecbf049c4181dca (127.0.98.3:43111)
I20260924 04:47:15.031225 7920 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54099
I20260924 04:47:15.033169 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:47:15.037902 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.031020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46697 (local address 127.0.98.62:43703)
0924 04:47:15.031111 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.031114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.031357 (+ 243us) server_negotiation.cc:407] Connection header received
0924 04:47:15.031434 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.031436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.031489 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.031614 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:15.032171 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.032719 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.033627 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.033843 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.033926 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.034964 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:15.034989 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:15.034992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:15.035016 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:15.036380 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:15.036823 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:15.036826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:15.036828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:15.036905 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:15.037143 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:15.037146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:15.037148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:15.037322 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:15.037404 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.037650 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.037752 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 04:47:15.037938 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.030916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.98.3:46697)
0924 04:47:15.031250 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:15.031264 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:15.031357 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:15.031711 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:15.031713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:15.031904 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:15.032031 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.032039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.032834 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.032837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.033518 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.033536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.033935 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.033939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.034026 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.034496 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:15.034507 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:15.034844 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:15.036492 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:15.036496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:15.036498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:15.036734 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:15.036984 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:15.036987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:15.036989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:15.037067 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:15.037405 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:15.037408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:15.037499 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 04:47:15.037590 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 04:47:15.037742 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
Sep 24 04:47:15 dist-test-slave-hvcl krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:47:15 dist-test-slave-hvcl krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:15 dist-test-slave-hvcl krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:47:15.056876 7970 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:15.050450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.0.98.62:43703)
0924 04:47:15.050578 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.050581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.050592 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:47:15.050630 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.050632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.050669 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.050734 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:15.051578 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.052102 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.052922 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.053140 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.053226 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.056723 (+ 3497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44928: BlockingRecv error: recv got EOF from 127.0.0.1:44928 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 04:47:15 dist-test-slave-hvcl krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:47:15.066249 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.058105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.0.98.62:43703)
0924 04:47:15.058214 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.058218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.058377 (+ 159us) server_negotiation.cc:407] Connection header received
0924 04:47:15.058426 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.058428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.058619 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.058700 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:15.059363 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.059945 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.060799 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.061008 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.061084 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.063344 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:15.063348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:15.063350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:15.063375 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:15.064716 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:15.065111 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:15.065113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:15.065115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:15.065192 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:15.065451 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:15.065454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:15.065456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:15.065654 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:15.065769 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.066009 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.066129 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:47:15.069188 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:47:15.072196 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:47:15.086472 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.081306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.0.98.3:43111)
0924 04:47:15.081727 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.081737 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.082015 (+ 278us) server_negotiation.cc:407] Connection header received
0924 04:47:15.082097 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.082103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.082319 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.082467 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:15.082669 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.083580 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.084749 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.085445 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.085518 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.085663 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.086189 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.086289 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":111,"threads_started":1}
I20260924 04:47:15.086460 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.081038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43111 (local address 127.0.0.1:51442)
0924 04:47:15.081905 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:15.081918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:15.082006 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:15.082446 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:15.082449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:15.082464 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:15.082570 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.082576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.083681 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.083684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.084606 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.084614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.085743 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.085745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.085951 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.086026 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 04:47:15.086140 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 04:47:15.086285 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":105,"thread_start_us":92,"threads_started":1}
I20260924 04:47:15.087023 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.080610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.0.98.2:42155)
0924 04:47:15.080918 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.080930 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.080955 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:47:15.081116 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.081123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.081339 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.081513 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:15.081974 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.083124 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.084758 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.085603 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.085704 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.085858 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.086757 (+ 899us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.086854 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":97,"threads_started":1}
I20260924 04:47:15.087078 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.080409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42155 (local address 127.0.0.1:34812)
0924 04:47:15.080765 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:15.080821 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:15.080966 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:15.081688 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:15.081694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:15.081716 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:15.081847 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.081853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.083227 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.083230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.084627 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.084634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.085716 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.085720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.085932 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.086026 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 04:47:15.086168 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 04:47:15.086402 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":52,"thread_start_us":79,"threads_started":1}
I20260924 04:47:15.087277 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.081120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33507 (local address 127.0.0.1:38408)
0924 04:47:15.082168 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:15.082177 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:15.082253 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:15.083072 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:15.083074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:15.083089 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:15.083165 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.083169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.084540 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.084543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.085733 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.085742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.086830 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.086833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.086938 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.086960 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:47:15.087049 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 04:47:15.087122 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":63,"threads_started":1}
I20260924 04:47:15.087651 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.081556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.0.98.1:33507)
0924 04:47:15.081893 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.081919 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.082255 (+ 336us) server_negotiation.cc:407] Connection header received
0924 04:47:15.082350 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.082355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.082539 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.082683 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:15.083282 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.084419 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.085876 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.086547 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.086645 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.086775 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.087430 (+ 655us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.087529 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":92,"threads_started":1}
I20260924 04:47:15.089722 8198 tablet_service.cc:1467] Processing CreateTablet for tablet 40ef4fec8cfb47048eb6b60cd06642f7 (DEFAULT_TABLE table=test-table [id=448508724eef43f5a358f8d74d12e5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:15.089741 8338 tablet_service.cc:1467] Processing CreateTablet for tablet 40ef4fec8cfb47048eb6b60cd06642f7 (DEFAULT_TABLE table=test-table [id=448508724eef43f5a358f8d74d12e5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:15.090674 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ef4fec8cfb47048eb6b60cd06642f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.090385 8058 tablet_service.cc:1467] Processing CreateTablet for tablet 40ef4fec8cfb47048eb6b60cd06642f7 (DEFAULT_TABLE table=test-table [id=448508724eef43f5a358f8d74d12e5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:15.090763 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ef4fec8cfb47048eb6b60cd06642f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.091295 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ef4fec8cfb47048eb6b60cd06642f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.097031 8432 tablet_bootstrap.cc:492] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222: Bootstrap starting.
I20260924 04:47:15.098580 8433 tablet_bootstrap.cc:492] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653: Bootstrap starting.
I20260924 04:47:15.098580 8434 tablet_bootstrap.cc:492] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca: Bootstrap starting.
I20260924 04:47:15.099752 8432 tablet_bootstrap.cc:654] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.100415 8432 log.cc:824] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:15.100585 8434 tablet_bootstrap.cc:654] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.100586 8433 tablet_bootstrap.cc:654] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.101369 8433 log.cc:824] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:15.101369 8434 log.cc:824] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:15.102164 8432 tablet_bootstrap.cc:492] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222: No bootstrap required, opened a new log
I20260924 04:47:15.102336 8432 ts_tablet_manager.cc:1397] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:47:15.103206 8433 tablet_bootstrap.cc:492] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653: No bootstrap required, opened a new log
I20260924 04:47:15.103206 8434 tablet_bootstrap.cc:492] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca: No bootstrap required, opened a new log
I20260924 04:47:15.103381 8433 ts_tablet_manager.cc:1397] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:47:15.103381 8434 ts_tablet_manager.cc:1397] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:47:15.105418 8432 raft_consensus.cc:348] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.105636 8432 raft_consensus.cc:374] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.105685 8432 raft_consensus.cc:729] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dfbe556b6874ef9ad97379390ccc222, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.106076 8432 consensus_queue.cc:260] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.106678 8264 heartbeater.cc:503] Master 127.0.98.62:43703 was elected leader, sending a full tablet report...
I20260924 04:47:15.107087 8432 ts_tablet_manager.cc:1428] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:47:15.107303 8433 raft_consensus.cc:348] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.107303 8434 raft_consensus.cc:348] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.107543 8433 raft_consensus.cc:374] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.107543 8434 raft_consensus.cc:374] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.107591 8434 raft_consensus.cc:729] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada18d69250940839ecbf049c4181dca, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.107591 8433 raft_consensus.cc:729] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9c3ee360cd4a28ac4715a826332653, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.108016 8434 consensus_queue.cc:260] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.108016 8433 consensus_queue.cc:260] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.108614 8124 heartbeater.cc:503] Master 127.0.98.62:43703 was elected leader, sending a full tablet report...
I20260924 04:47:15.108958 8434 ts_tablet_manager.cc:1428] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:47:15.109014 8433 ts_tablet_manager.cc:1428] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:47:15.109242 8404 heartbeater.cc:503] Master 127.0.98.62:43703 was elected leader, sending a full tablet report...
I20260924 04:47:15.235610 8439 raft_consensus.cc:482] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:47:15.235821 8439 raft_consensus.cc:504] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.236956 8439 leader_election.cc:288] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155), ada18d69250940839ecbf049c4181dca (127.0.98.3:43111)
I20260924 04:47:15.240713 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.237356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58019 (local address 127.0.98.3:43111)
0924 04:47:15.237488 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.237493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.237663 (+ 170us) server_negotiation.cc:407] Connection header received
0924 04:47:15.237803 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.237807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.237850 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.237911 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:15.238181 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.238685 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.239812 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.240202 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.240293 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.240356 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.240523 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.240589 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:47:15.240775 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.237381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35799 (local address 127.0.98.2:42155)
0924 04:47:15.237506 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.237511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.237878 (+ 367us) server_negotiation.cc:407] Connection header received
0924 04:47:15.238002 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.238004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.238045 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.238105 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:15.238340 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.238843 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.239890 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.240267 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.240356 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.240415 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.240590 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.240656 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:47:15.240795 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.237238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42155 (local address 127.0.98.1:35799)
0924 04:47:15.237787 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:15.237799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:15.237896 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:15.238122 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:15.238125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:15.238140 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:15.238243 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.238249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.238942 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.238945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.239773 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.239782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.240374 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.240378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.240489 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.240506 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:47:15.240566 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:47:15.240633 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":302,"thread_start_us":104,"threads_started":1}
I20260924 04:47:15.240935 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.237231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43111 (local address 127.0.98.1:58019)
0924 04:47:15.237574 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:15.237586 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:15.237699 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:15.237936 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:15.237939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:15.237954 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:15.238077 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.238083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.238790 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.238793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.239692 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.239701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.240308 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.240311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.240409 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.240432 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:47:15.240497 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:47:15.240572 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":19,"thread_start_us":113,"threads_started":1}
I20260924 04:47:15.241408 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ef4fec8cfb47048eb6b60cd06642f7" candidate_uuid: "7e9c3ee360cd4a28ac4715a826332653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dfbe556b6874ef9ad97379390ccc222" is_pre_election: true
I20260924 04:47:15.241421 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ef4fec8cfb47048eb6b60cd06642f7" candidate_uuid: "7e9c3ee360cd4a28ac4715a826332653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada18d69250940839ecbf049c4181dca" is_pre_election: true
I20260924 04:47:15.241700 8358 raft_consensus.cc:2454] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c3ee360cd4a28ac4715a826332653 in term 0.
I20260924 04:47:15.241700 8218 raft_consensus.cc:2454] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c3ee360cd4a28ac4715a826332653 in term 0.
I20260924 04:47:15.242127 8009 leader_election.cc:302] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dfbe556b6874ef9ad97379390ccc222, 7e9c3ee360cd4a28ac4715a826332653; no voters:
I20260924 04:47:15.242388 8439 raft_consensus.cc:2781] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:47:15.242465 8439 raft_consensus.cc:482] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:47:15.242496 8439 raft_consensus.cc:3037] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:15.243613 8439 raft_consensus.cc:504] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.243963 8439 leader_election.cc:288] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [CANDIDATE]: Term 1 election: Requested vote from peers 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155), ada18d69250940839ecbf049c4181dca (127.0.98.3:43111)
I20260924 04:47:15.244299 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ef4fec8cfb47048eb6b60cd06642f7" candidate_uuid: "7e9c3ee360cd4a28ac4715a826332653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dfbe556b6874ef9ad97379390ccc222"
I20260924 04:47:15.244302 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ef4fec8cfb47048eb6b60cd06642f7" candidate_uuid: "7e9c3ee360cd4a28ac4715a826332653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada18d69250940839ecbf049c4181dca"
I20260924 04:47:15.244418 8218 raft_consensus.cc:3037] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:15.244426 8358 raft_consensus.cc:3037] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:15.245502 8358 raft_consensus.cc:2454] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c3ee360cd4a28ac4715a826332653 in term 1.
I20260924 04:47:15.245795 8012 leader_election.cc:302] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c3ee360cd4a28ac4715a826332653, ada18d69250940839ecbf049c4181dca; no voters:
I20260924 04:47:15.245918 8218 raft_consensus.cc:2454] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c3ee360cd4a28ac4715a826332653 in term 1.
I20260924 04:47:15.245973 8439 raft_consensus.cc:2781] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:15.246196 8439 raft_consensus.cc:686] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [term 1 LEADER]: Becoming Leader. State: Replica: 7e9c3ee360cd4a28ac4715a826332653, State: Running, Role: LEADER
I20260924 04:47:15.246529 8439 consensus_queue.cc:237] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.249709 7920 catalog_manager.cc:5495] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e9c3ee360cd4a28ac4715a826332653 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7e9c3ee360cd4a28ac4715a826332653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } health_report { overall_health: UNKNOWN } } }
W20260924 04:47:15.275890 8405 tablet.cc:2369] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:47:15.278355 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.275008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.0.98.1:33507)
0924 04:47:15.275132 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.275137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.275151 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:47:15.275261 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.275263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.275319 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.275405 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:47:15.275634 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.276154 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.276992 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.277211 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.277294 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.277584 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:47:15.278004 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:47:15.278104 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.278173 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.278226 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:47:15.283869 8358 raft_consensus.cc:1261] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Refusing update from remote peer 7e9c3ee360cd4a28ac4715a826332653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:15.283870 8218 raft_consensus.cc:1261] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Refusing update from remote peer 7e9c3ee360cd4a28ac4715a826332653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:15.284538 8443 consensus_queue.cc:1035] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:15.284706 8439 consensus_queue.cc:1035] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:15.293248 8447 mvcc.cc:205] Tried to move back new op lower bound from 7332762563716059136 to 7332762563572281344. Current Snapshot: MvccSnapshot[applied={T|T < 7332762563716059136}]
I20260924 04:47:15.301101 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:15.301266 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:15.303680 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6: Sending DeleteTablet for 3 replicas of tablet 40ef4fec8cfb47048eb6b60cd06642f7
I20260924 04:47:15.304549 8198 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ef4fec8cfb47048eb6b60cd06642f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34812
I20260924 04:47:15.304553 8338 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ef4fec8cfb47048eb6b60cd06642f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51442
I20260924 04:47:15.304551 8058 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ef4fec8cfb47048eb6b60cd06642f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38408
I20260924 04:47:15.305084 8458 tablet_replica.cc:330] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653: stopping tablet replica
I20260924 04:47:15.305102 8456 tablet_replica.cc:330] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222: stopping tablet replica
I20260924 04:47:15.305478 8458 raft_consensus.cc:2229] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:47:15.305858 8456 raft_consensus.cc:2229] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:15.305824 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:47:15.306157 8458 raft_consensus.cc:2258] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:15.306170 8456 raft_consensus.cc:2258] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 04:47:15.306262 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:47:15.306706 8457 tablet_replica.cc:330] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca: stopping tablet replica
I20260924 04:47:15.307813 8458 ts_tablet_manager.cc:1905] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:15.308133 8456 ts_tablet_manager.cc:1905] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:15.310735 8458 ts_tablet_manager.cc:1918] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:15.310896 8458 log.cc:1196] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/wal/wals/40ef4fec8cfb47048eb6b60cd06642f7
I20260924 04:47:15.311069 8457 raft_consensus.cc:2229] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:15.311352 8458 ts_tablet_manager.cc:1939] T 40ef4fec8cfb47048eb6b60cd06642f7 P 7e9c3ee360cd4a28ac4715a826332653: Deleting consensus metadata
I20260924 04:47:15.311451 8457 raft_consensus.cc:2258] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:15.310962 8456 ts_tablet_manager.cc:1918] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:15.311811 8456 log.cc:1196] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/40ef4fec8cfb47048eb6b60cd06642f7
I20260924 04:47:15.312103 8456 ts_tablet_manager.cc:1939] T 40ef4fec8cfb47048eb6b60cd06642f7 P 6dfbe556b6874ef9ad97379390ccc222: Deleting consensus metadata
I20260924 04:47:15.312076 8198 tablet_service.cc:1467] Processing CreateTablet for tablet 569a92a8b6e746c3b86b968457e1f8d1 (DEFAULT_TABLE table=test-table [id=d0912c09cb2a4f559b88035cb4303fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:15.312317 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 569a92a8b6e746c3b86b968457e1f8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.313287 7905 catalog_manager.cc:4841] TS 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155): tablet 40ef4fec8cfb47048eb6b60cd06642f7 (table test-table [id=448508724eef43f5a358f8d74d12e5c8]) successfully deleted
I20260924 04:47:15.314476 8432 tablet_bootstrap.cc:492] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222: Bootstrap starting.
I20260924 04:47:15.314553 8457 ts_tablet_manager.cc:1905] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:15.315021 8338 tablet_service.cc:1467] Processing CreateTablet for tablet 569a92a8b6e746c3b86b968457e1f8d1 (DEFAULT_TABLE table=test-table [id=d0912c09cb2a4f559b88035cb4303fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:15.315244 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 569a92a8b6e746c3b86b968457e1f8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.315416 8432 tablet_bootstrap.cc:654] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.317023 8457 ts_tablet_manager.cc:1918] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:15.317186 8434 tablet_bootstrap.cc:492] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca: Bootstrap starting.
I20260924 04:47:15.317739 7905 catalog_manager.cc:4841] TS 7e9c3ee360cd4a28ac4715a826332653 (127.0.98.1:33507): tablet 40ef4fec8cfb47048eb6b60cd06642f7 (table test-table [id=448508724eef43f5a358f8d74d12e5c8]) successfully deleted
I20260924 04:47:15.318081 8434 tablet_bootstrap.cc:654] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.318100 8058 tablet_service.cc:1467] Processing CreateTablet for tablet 569a92a8b6e746c3b86b968457e1f8d1 (DEFAULT_TABLE table=test-table [id=d0912c09cb2a4f559b88035cb4303fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:15.318359 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 569a92a8b6e746c3b86b968457e1f8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.318598 8457 log.cc:1196] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/40ef4fec8cfb47048eb6b60cd06642f7
I20260924 04:47:15.318895 8457 ts_tablet_manager.cc:1939] T 40ef4fec8cfb47048eb6b60cd06642f7 P ada18d69250940839ecbf049c4181dca: Deleting consensus metadata
I20260924 04:47:15.319300 8434 tablet_bootstrap.cc:492] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca: No bootstrap required, opened a new log
I20260924 04:47:15.319372 8434 ts_tablet_manager.cc:1397] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:15.319792 8434 raft_consensus.cc:348] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.319895 8434 raft_consensus.cc:374] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.319923 8434 raft_consensus.cc:729] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada18d69250940839ecbf049c4181dca, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.320040 8434 consensus_queue.cc:260] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.320147 7908 catalog_manager.cc:4841] TS ada18d69250940839ecbf049c4181dca (127.0.98.3:43111): tablet 40ef4fec8cfb47048eb6b60cd06642f7 (table test-table [id=448508724eef43f5a358f8d74d12e5c8]) successfully deleted
I20260924 04:47:15.320271 8434 ts_tablet_manager.cc:1428] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:15.320596 8433 tablet_bootstrap.cc:492] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653: Bootstrap starting.
I20260924 04:47:15.321520 8433 tablet_bootstrap.cc:654] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.322954 8433 tablet_bootstrap.cc:492] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653: No bootstrap required, opened a new log
I20260924 04:47:15.323010 8432 tablet_bootstrap.cc:492] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222: No bootstrap required, opened a new log
I20260924 04:47:15.323060 8433 ts_tablet_manager.cc:1397] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:47:15.323082 8432 ts_tablet_manager.cc:1397] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260924 04:47:15.323494 8432 raft_consensus.cc:348] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.323494 8433 raft_consensus.cc:348] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.323606 8432 raft_consensus.cc:374] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.323606 8433 raft_consensus.cc:374] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.323637 8433 raft_consensus.cc:729] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9c3ee360cd4a28ac4715a826332653, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.323637 8432 raft_consensus.cc:729] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dfbe556b6874ef9ad97379390ccc222, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.323755 8432 consensus_queue.cc:260] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.323755 8433 consensus_queue.cc:260] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.324003 8432 ts_tablet_manager.cc:1428] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:15.324007 8433 ts_tablet_manager.cc:1428] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260924 04:47:15.331549 8125 tablet.cc:2369] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:47:15.346843 8440 raft_consensus.cc:482] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:47:15.347014 8440 raft_consensus.cc:504] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.348232 8440 leader_election.cc:288] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e9c3ee360cd4a28ac4715a826332653 (127.0.98.1:33507), 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155)
I20260924 04:47:15.352380 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.348385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33507 (local address 127.0.98.3:47191)
0924 04:47:15.348524 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:15.348541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:15.348629 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:15.349099 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:15.349102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:15.349132 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:15.349241 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.349247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.350053 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.350056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.351054 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.351064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.352000 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.352004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.352099 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.352125 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:47:15.352188 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:47:15.352264 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"spinlock_wait_cycles":6272}
I20260924 04:47:15.352617 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.348901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54703 (local address 127.0.98.2:42155)
0924 04:47:15.349017 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.349022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.349080 (+ 58us) server_negotiation.cc:407] Connection header received
0924 04:47:15.349222 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.349225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.349261 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.349319 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:15.349670 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.350435 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.351641 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.352079 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.352176 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.352233 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.352429 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.352507 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:47:15.352733 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.348703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47191 (local address 127.0.98.1:33507)
0924 04:47:15.348821 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.348826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.348839 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:47:15.348886 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.348889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.348931 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.348993 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:15.349357 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.349959 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.351170 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.351886 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.351995 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.352063 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.352558 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.352631 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:47:15.353102 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.348653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42155 (local address 127.0.98.3:54703)
0924 04:47:15.349008 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:15.349022 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:15.349130 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:15.349419 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:15.349422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:15.349437 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:15.349546 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.349552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.350556 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.350560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.351527 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.351535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.352202 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.352206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.352311 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.352333 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:47:15.352399 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:47:15.352953 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"spinlock_wait_cycles":1280,"thread_start_us":65,"threads_started":1}
I20260924 04:47:15.353232 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569a92a8b6e746c3b86b968457e1f8d1" candidate_uuid: "ada18d69250940839ecbf049c4181dca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9c3ee360cd4a28ac4715a826332653" is_pre_election: true
I20260924 04:47:15.353480 8078 raft_consensus.cc:2454] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada18d69250940839ecbf049c4181dca in term 0.
I20260924 04:47:15.353904 8289 leader_election.cc:302] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c3ee360cd4a28ac4715a826332653, ada18d69250940839ecbf049c4181dca; no voters:
I20260924 04:47:15.353995 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569a92a8b6e746c3b86b968457e1f8d1" candidate_uuid: "ada18d69250940839ecbf049c4181dca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dfbe556b6874ef9ad97379390ccc222" is_pre_election: true
I20260924 04:47:15.354133 8218 raft_consensus.cc:2454] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada18d69250940839ecbf049c4181dca in term 0.
I20260924 04:47:15.354167 8440 raft_consensus.cc:2781] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:47:15.354213 8440 raft_consensus.cc:482] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:47:15.354257 8440 raft_consensus.cc:3037] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:15.355428 8440 raft_consensus.cc:504] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.355759 8440 leader_election.cc:288] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [CANDIDATE]: Term 1 election: Requested vote from peers 7e9c3ee360cd4a28ac4715a826332653 (127.0.98.1:33507), 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155)
I20260924 04:47:15.356148 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569a92a8b6e746c3b86b968457e1f8d1" candidate_uuid: "ada18d69250940839ecbf049c4181dca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dfbe556b6874ef9ad97379390ccc222"
I20260924 04:47:15.356161 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569a92a8b6e746c3b86b968457e1f8d1" candidate_uuid: "ada18d69250940839ecbf049c4181dca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9c3ee360cd4a28ac4715a826332653"
I20260924 04:47:15.356273 8078 raft_consensus.cc:3037] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:15.356281 8218 raft_consensus.cc:3037] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:15.357376 8078 raft_consensus.cc:2454] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada18d69250940839ecbf049c4181dca in term 1.
I20260924 04:47:15.357376 8218 raft_consensus.cc:2454] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada18d69250940839ecbf049c4181dca in term 1.
I20260924 04:47:15.357635 8289 leader_election.cc:302] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dfbe556b6874ef9ad97379390ccc222, ada18d69250940839ecbf049c4181dca; no voters:
I20260924 04:47:15.357800 8440 raft_consensus.cc:2781] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:15.358021 8440 raft_consensus.cc:686] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [term 1 LEADER]: Becoming Leader. State: Replica: ada18d69250940839ecbf049c4181dca, State: Running, Role: LEADER
I20260924 04:47:15.358273 8440 consensus_queue.cc:237] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.360426 7921 catalog_manager.cc:5495] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca reported cstate change: term changed from 0 to 1, leader changed from <none> to ada18d69250940839ecbf049c4181dca (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ada18d69250940839ecbf049c4181dca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } health_report { overall_health: HEALTHY } } }
Sep 24 04:47:15 dist-test-slave-hvcl krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225234, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:47:15.394310 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.384832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.0.98.62:43703)
0924 04:47:15.384961 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.384965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.385140 (+ 175us) server_negotiation.cc:407] Connection header received
0924 04:47:15.385264 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.385267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.385305 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.385369 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:15.386258 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.386922 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.388104 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.388356 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.388441 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.390923 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:15.390926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:15.390928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:15.390948 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:15.392403 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:15.392925 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:15.392928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:15.392929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:15.393013 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:15.393277 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:15.393279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:15.393280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:15.393507 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:15.393584 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.393930 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.394178 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":53}
I20260924 04:47:15.394313 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.384763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43703 (local address 127.0.0.1:44942)
0924 04:47:15.385055 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:15.385069 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:15.385186 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:15.385409 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:15.385412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:15.385999 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:15.386124 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.386131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.387060 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.387064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.387974 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.387983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.388460 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.388463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.388553 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.389116 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:15.389128 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:15.390777 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:15.392517 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:15.392527 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:15.392533 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:15.392818 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:15.393101 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:15.393104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:15.393106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:15.393198 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:15.393601 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:15.393607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:15.393728 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 04:47:15.393893 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 04:47:15.394095 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
I20260924 04:47:15.398285 7921 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44942:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:47:15.399087 7921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:47:15.403695 8058 tablet_service.cc:1467] Processing CreateTablet for tablet 39d781b913864e8f9ddabe82c9b155ee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1676c03384b9439db54c30fff3080224]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:47:15.403759 8198 tablet_service.cc:1467] Processing CreateTablet for tablet 39d781b913864e8f9ddabe82c9b155ee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1676c03384b9439db54c30fff3080224]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:47:15.404054 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d781b913864e8f9ddabe82c9b155ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.404110 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d781b913864e8f9ddabe82c9b155ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.406212 8433 tablet_bootstrap.cc:492] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653: Bootstrap starting.
I20260924 04:47:15.406540 8338 tablet_service.cc:1467] Processing CreateTablet for tablet 39d781b913864e8f9ddabe82c9b155ee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1676c03384b9439db54c30fff3080224]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:47:15.406885 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d781b913864e8f9ddabe82c9b155ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.407095 8432 tablet_bootstrap.cc:492] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222: Bootstrap starting.
I20260924 04:47:15.407972 8432 tablet_bootstrap.cc:654] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.408981 8433 tablet_bootstrap.cc:654] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.409986 8434 tablet_bootstrap.cc:492] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca: Bootstrap starting.
I20260924 04:47:15.410661 8433 tablet_bootstrap.cc:492] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653: No bootstrap required, opened a new log
I20260924 04:47:15.410756 8433 ts_tablet_manager.cc:1397] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260924 04:47:15.410816 8434 tablet_bootstrap.cc:654] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.411224 8433 raft_consensus.cc:348] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.411356 8433 raft_consensus.cc:374] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.411393 8433 raft_consensus.cc:729] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9c3ee360cd4a28ac4715a826332653, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.411523 8433 consensus_queue.cc:260] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.411780 8433 ts_tablet_manager.cc:1428] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:15.411738 8439 tablet_replica.cc:439] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } } }
I20260924 04:47:15.411854 8439 tablet_replica.cc:442] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:47:15.412331 8432 tablet_bootstrap.cc:492] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222: No bootstrap required, opened a new log
I20260924 04:47:15.412444 8432 ts_tablet_manager.cc:1397] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 04:47:15.412896 8432 raft_consensus.cc:348] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.413005 8432 raft_consensus.cc:374] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.413036 8432 raft_consensus.cc:729] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dfbe556b6874ef9ad97379390ccc222, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.413159 8432 consensus_queue.cc:260] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.413275 8434 tablet_bootstrap.cc:492] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca: No bootstrap required, opened a new log
I20260924 04:47:15.413362 8434 ts_tablet_manager.cc:1397] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:47:15.413408 8432 ts_tablet_manager.cc:1428] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:15.413750 8438 tablet_replica.cc:439] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } } }
I20260924 04:47:15.413806 8434 raft_consensus.cc:348] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.413882 8438 tablet_replica.cc:442] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:47:15.413928 8434 raft_consensus.cc:374] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.413959 8434 raft_consensus.cc:729] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada18d69250940839ecbf049c4181dca, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.414068 8434 consensus_queue.cc:260] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.414317 8434 ts_tablet_manager.cc:1428] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:47:15.415369 8440 tablet_replica.cc:439] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } } }
I20260924 04:47:15.415525 8440 tablet_replica.cc:442] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 04:47:15.555979 8265 tablet.cc:2369] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:47:15.672976 8440 raft_consensus.cc:482] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:47:15.673111 8440 raft_consensus.cc:504] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.673544 8440 leader_election.cc:288] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e9c3ee360cd4a28ac4715a826332653 (127.0.98.1:33507), 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155)
I20260924 04:47:15.674001 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d781b913864e8f9ddabe82c9b155ee" candidate_uuid: "ada18d69250940839ecbf049c4181dca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9c3ee360cd4a28ac4715a826332653" is_pre_election: true
I20260924 04:47:15.674146 8078 raft_consensus.cc:2454] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada18d69250940839ecbf049c4181dca in term 0.
I20260924 04:47:15.674075 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d781b913864e8f9ddabe82c9b155ee" candidate_uuid: "ada18d69250940839ecbf049c4181dca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dfbe556b6874ef9ad97379390ccc222" is_pre_election: true
I20260924 04:47:15.674268 8218 raft_consensus.cc:2454] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada18d69250940839ecbf049c4181dca in term 0.
I20260924 04:47:15.674487 8289 leader_election.cc:302] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c3ee360cd4a28ac4715a826332653, ada18d69250940839ecbf049c4181dca; no voters:
I20260924 04:47:15.674645 8440 raft_consensus.cc:2781] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:47:15.674705 8440 raft_consensus.cc:482] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:47:15.674749 8440 raft_consensus.cc:3037] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:15.675810 8440 raft_consensus.cc:504] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.676108 8440 leader_election.cc:288] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [CANDIDATE]: Term 1 election: Requested vote from peers 7e9c3ee360cd4a28ac4715a826332653 (127.0.98.1:33507), 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155)
I20260924 04:47:15.676535 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d781b913864e8f9ddabe82c9b155ee" candidate_uuid: "ada18d69250940839ecbf049c4181dca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9c3ee360cd4a28ac4715a826332653"
I20260924 04:47:15.676555 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d781b913864e8f9ddabe82c9b155ee" candidate_uuid: "ada18d69250940839ecbf049c4181dca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dfbe556b6874ef9ad97379390ccc222"
I20260924 04:47:15.676669 8218 raft_consensus.cc:3037] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:15.676669 8078 raft_consensus.cc:3037] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:15.677798 8078 raft_consensus.cc:2454] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada18d69250940839ecbf049c4181dca in term 1.
I20260924 04:47:15.677798 8218 raft_consensus.cc:2454] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada18d69250940839ecbf049c4181dca in term 1.
I20260924 04:47:15.678097 8289 leader_election.cc:302] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dfbe556b6874ef9ad97379390ccc222, ada18d69250940839ecbf049c4181dca; no voters:
I20260924 04:47:15.678261 8440 raft_consensus.cc:2781] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:15.678347 8440 raft_consensus.cc:686] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [term 1 LEADER]: Becoming Leader. State: Replica: ada18d69250940839ecbf049c4181dca, State: Running, Role: LEADER
I20260924 04:47:15.678499 8440 consensus_queue.cc:237] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.679203 8463 tablet_replica.cc:439] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca: TxnStatusTablet state changed. Reason: New leader ada18d69250940839ecbf049c4181dca. Latest consensus state: current_term: 1 leader_uuid: "ada18d69250940839ecbf049c4181dca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } } }
I20260924 04:47:15.679299 8463 tablet_replica.cc:442] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:47:15.679517 8475 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:47:15.679692 8475 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:47:15.680238 7921 catalog_manager.cc:5495] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca reported cstate change: term changed from 0 to 1, leader changed from <none> to ada18d69250940839ecbf049c4181dca (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ada18d69250940839ecbf049c4181dca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } health_report { overall_health: HEALTHY } } }
I20260924 04:47:15.707612 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.704183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.0.98.3:43111)
0924 04:47:15.704296 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.704300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.704315 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:47:15.704476 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.704479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.704532 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.704619 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:47:15.704850 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.705373 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.706201 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.706413 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.706496 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.706771 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:47:15.707233 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:47:15.707332 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.707404 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.707458 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:47:15.707610 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.704071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43111 (local address 127.0.0.1:53202)
0924 04:47:15.704216 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:15.704230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:15.704335 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:15.704640 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:15.704643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:15.704654 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:47:15.704752 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.704757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.705472 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.705475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.706088 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.706096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.706504 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.706508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.706592 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.706636 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:47:15.707314 (+ 678us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:47:15.707330 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 04:47:15.707394 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:47:15.707481 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 04:47:15.710865 8218 raft_consensus.cc:1261] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Refusing update from remote peer ada18d69250940839ecbf049c4181dca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:15.711426 8078 raft_consensus.cc:1261] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Refusing update from remote peer ada18d69250940839ecbf049c4181dca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:15.712028 8440 consensus_queue.cc:1035] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:15.712273 8463 consensus_queue.cc:1035] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:15.714605 8438 tablet_replica.cc:439] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222: TxnStatusTablet state changed. Reason: New leader ada18d69250940839ecbf049c4181dca. Latest consensus state: current_term: 1 leader_uuid: "ada18d69250940839ecbf049c4181dca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } } }
I20260924 04:47:15.714738 8438 tablet_replica.cc:442] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:47:15.715420 8439 tablet_replica.cc:439] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653: TxnStatusTablet state changed. Reason: New leader ada18d69250940839ecbf049c4181dca. Latest consensus state: current_term: 1 leader_uuid: "ada18d69250940839ecbf049c4181dca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } } }
I20260924 04:47:15.715536 8439 tablet_replica.cc:442] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:47:15.715834 8440 tablet_replica.cc:439] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ada18d69250940839ecbf049c4181dca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } } }
I20260924 04:47:15.715942 8440 tablet_replica.cc:442] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:47:15.716028 8448 mvcc.cc:205] Tried to move back new op lower bound from 7332762565465964544 to 7332762565340545024. Current Snapshot: MvccSnapshot[applied={T|T < 7332762565465964544}]
I20260924 04:47:15.717995 8439 tablet_replica.cc:439] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ada18d69250940839ecbf049c4181dca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } } }
I20260924 04:47:15.718101 8439 tablet_replica.cc:442] T 39d781b913864e8f9ddabe82c9b155ee P 7e9c3ee360cd4a28ac4715a826332653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:47:15.720014 8440 tablet_replica.cc:439] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ada18d69250940839ecbf049c4181dca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } } }
I20260924 04:47:15.720114 8440 tablet_replica.cc:442] T 39d781b913864e8f9ddabe82c9b155ee P ada18d69250940839ecbf049c4181dca: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:47:15.721870 8438 tablet_replica.cc:439] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ada18d69250940839ecbf049c4181dca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } } }
I20260924 04:47:15.721978 8438 tablet_replica.cc:442] T 39d781b913864e8f9ddabe82c9b155ee P 6dfbe556b6874ef9ad97379390ccc222: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:47:15.724325 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.720974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.0.98.3:43111)
0924 04:47:15.721085 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.721090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.721104 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:47:15.721142 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.721144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.721198 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.721266 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:47:15.721572 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.722153 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.723409 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.723629 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.723718 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.723943 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:47:15.724011 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:47:15.724067 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.724162 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.724207 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:47:15.734057 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.731135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39673 (local address 127.0.98.3:43111)
0924 04:47:15.731237 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.731242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.731360 (+ 118us) server_negotiation.cc:407] Connection header received
0924 04:47:15.731471 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.731473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.731513 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.731576 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:47:15.731778 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.732247 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.733054 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.733302 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.733380 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.733632 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:47:15.733693 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:47:15.733745 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.733826 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.733889 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":42}
I20260924 04:47:15.734073 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.731042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43111 (local address 127.0.98.3:39673)
0924 04:47:15.731293 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:15.731304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:15.731393 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:15.731589 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:15.731591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:15.731598 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:47:15.731698 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.731703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.732333 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.732336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.732956 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:15.732963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.733386 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.733389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:15.733474 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.733521 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:47:15.733757 (+ 236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:47:15.733762 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:47:15.733818 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 04:47:15.733901 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260924 04:47:15.740691 8218 raft_consensus.cc:1261] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Refusing update from remote peer ada18d69250940839ecbf049c4181dca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:15.740736 8078 raft_consensus.cc:1261] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Refusing update from remote peer ada18d69250940839ecbf049c4181dca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:15.741076 8463 consensus_queue.cc:1035] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:15.741276 8440 consensus_queue.cc:1035] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:15.747376 8446 mvcc.cc:205] Tried to move back new op lower bound from 7332762565590908928 to 7332762564029558784. Current Snapshot: MvccSnapshot[applied={T|T < 7332762565590908928}]
I20260924 04:47:15.780476 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:15.780684 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:15.782646 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6: Sending DeleteTablet for 3 replicas of tablet 569a92a8b6e746c3b86b968457e1f8d1
I20260924 04:47:15.783303 8058 tablet_service.cc:1514] Processing DeleteTablet for tablet 569a92a8b6e746c3b86b968457e1f8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38408
I20260924 04:47:15.783324 8198 tablet_service.cc:1514] Processing DeleteTablet for tablet 569a92a8b6e746c3b86b968457e1f8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34812
I20260924 04:47:15.783496 8458 tablet_replica.cc:330] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653: stopping tablet replica
I20260924 04:47:15.783505 8456 tablet_replica.cc:330] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222: stopping tablet replica
I20260924 04:47:15.783612 8458 raft_consensus.cc:2229] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:15.783632 8456 raft_consensus.cc:2229] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:15.783730 8458 raft_consensus.cc:2258] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:15.783752 8456 raft_consensus.cc:2258] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:15.783914 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet 569a92a8b6e746c3b86b968457e1f8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51442
I20260924 04:47:15.784086 8457 tablet_replica.cc:330] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca: stopping tablet replica
I20260924 04:47:15.784197 8457 raft_consensus.cc:2229] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:47:15.784492 8457 raft_consensus.cc:2258] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:15.784526 8456 ts_tablet_manager.cc:1905] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:15.784551 8458 ts_tablet_manager.cc:1905] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:15.786852 8458 ts_tablet_manager.cc:1918] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:47:15.786913 8458 log.cc:1196] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/wal/wals/569a92a8b6e746c3b86b968457e1f8d1
I20260924 04:47:15.787189 8458 ts_tablet_manager.cc:1939] T 569a92a8b6e746c3b86b968457e1f8d1 P 7e9c3ee360cd4a28ac4715a826332653: Deleting consensus metadata
I20260924 04:47:15.787703 8457 ts_tablet_manager.cc:1905] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:15.788426 7905 catalog_manager.cc:4841] TS 7e9c3ee360cd4a28ac4715a826332653 (127.0.98.1:33507): tablet 569a92a8b6e746c3b86b968457e1f8d1 (table test-table [id=d0912c09cb2a4f559b88035cb4303fca]) successfully deleted
I20260924 04:47:15.790031 8457 ts_tablet_manager.cc:1918] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:47:15.790091 8457 log.cc:1196] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/569a92a8b6e746c3b86b968457e1f8d1
I20260924 04:47:15.790298 8457 ts_tablet_manager.cc:1939] T 569a92a8b6e746c3b86b968457e1f8d1 P ada18d69250940839ecbf049c4181dca: Deleting consensus metadata
I20260924 04:47:15.787765 8456 ts_tablet_manager.cc:1918] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:47:15.791445 8456 log.cc:1196] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/569a92a8b6e746c3b86b968457e1f8d1
I20260924 04:47:15.791726 8456 ts_tablet_manager.cc:1939] T 569a92a8b6e746c3b86b968457e1f8d1 P 6dfbe556b6874ef9ad97379390ccc222: Deleting consensus metadata
I20260924 04:47:15.792388 7908 catalog_manager.cc:4841] TS ada18d69250940839ecbf049c4181dca (127.0.98.3:43111): tablet 569a92a8b6e746c3b86b968457e1f8d1 (table test-table [id=d0912c09cb2a4f559b88035cb4303fca]) successfully deleted
I20260924 04:47:15.792430 7905 catalog_manager.cc:4841] TS 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155): tablet 569a92a8b6e746c3b86b968457e1f8d1 (table test-table [id=d0912c09cb2a4f559b88035cb4303fca]) successfully deleted
Sep 24 04:47:15 dist-test-slave-hvcl krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:47:15 dist-test-slave-hvcl krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:15 dist-test-slave-hvcl krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:47:15.812656 7970 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:15.806280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50316 (local address 127.0.98.62:43703)
0924 04:47:15.806415 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.806420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.806664 (+ 244us) server_negotiation.cc:407] Connection header received
0924 04:47:15.806872 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.806876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.806930 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.807046 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:15.807895 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.808414 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.809284 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.809526 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.809619 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.812532 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50316: BlockingRecv error: recv got EOF from 127.0.0.1:50316 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 04:47:15 dist-test-slave-hvcl krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:47:15.822186 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.814329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50322 (local address 127.0.98.62:43703)
0924 04:47:15.814439 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.814443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.814800 (+ 357us) server_negotiation.cc:407] Connection header received
0924 04:47:15.814905 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.814908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.814949 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.815046 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:15.815677 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.816189 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.817023 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.817241 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.817326 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.819507 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:15.819510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:15.819512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:15.819533 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:15.820755 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:15.821117 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:15.821121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:15.821123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:15.821194 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:15.821426 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:15.821429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:15.821431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:15.821605 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:15.821722 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.821936 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.822038 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:47:15.824888 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:47:15.825316 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:47:15.829696 8198 tablet_service.cc:1467] Processing CreateTablet for tablet 3426afd69f3a45e784e3e10721a1da57 (DEFAULT_TABLE table=test-table [id=99d59b12ce1a43abad4e93c374316005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:15.829850 8058 tablet_service.cc:1467] Processing CreateTablet for tablet 3426afd69f3a45e784e3e10721a1da57 (DEFAULT_TABLE table=test-table [id=99d59b12ce1a43abad4e93c374316005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:15.829980 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3426afd69f3a45e784e3e10721a1da57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.830183 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3426afd69f3a45e784e3e10721a1da57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.832078 8433 tablet_bootstrap.cc:492] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653: Bootstrap starting.
I20260924 04:47:15.832334 8432 tablet_bootstrap.cc:492] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222: Bootstrap starting.
I20260924 04:47:15.832320 8337 tablet_service.cc:1467] Processing CreateTablet for tablet 3426afd69f3a45e784e3e10721a1da57 (DEFAULT_TABLE table=test-table [id=99d59b12ce1a43abad4e93c374316005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:15.832580 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3426afd69f3a45e784e3e10721a1da57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.832962 8433 tablet_bootstrap.cc:654] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.833166 8432 tablet_bootstrap.cc:654] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.835019 8434 tablet_bootstrap.cc:492] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca: Bootstrap starting.
I20260924 04:47:15.835399 8432 tablet_bootstrap.cc:492] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222: No bootstrap required, opened a new log
I20260924 04:47:15.835489 8432 ts_tablet_manager.cc:1397] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:47:15.835748 8434 tablet_bootstrap.cc:654] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.835917 8432 raft_consensus.cc:348] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.836048 8432 raft_consensus.cc:374] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.836077 8432 raft_consensus.cc:729] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dfbe556b6874ef9ad97379390ccc222, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.836210 8432 consensus_queue.cc:260] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.836464 8432 ts_tablet_manager.cc:1428] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:15.836936 8434 tablet_bootstrap.cc:492] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca: No bootstrap required, opened a new log
I20260924 04:47:15.837016 8434 ts_tablet_manager.cc:1397] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:15.837162 8433 tablet_bootstrap.cc:492] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653: No bootstrap required, opened a new log
I20260924 04:47:15.837236 8433 ts_tablet_manager.cc:1397] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 04:47:15.837396 8434 raft_consensus.cc:348] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.837491 8434 raft_consensus.cc:374] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.837520 8434 raft_consensus.cc:729] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada18d69250940839ecbf049c4181dca, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.837594 8433 raft_consensus.cc:348] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.837642 8434 consensus_queue.cc:260] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.837688 8433 raft_consensus.cc:374] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.837718 8433 raft_consensus.cc:729] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9c3ee360cd4a28ac4715a826332653, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.837826 8433 consensus_queue.cc:260] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.837884 8434 ts_tablet_manager.cc:1428] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:15.838058 8433 ts_tablet_manager.cc:1428] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:15.896067 8439 raft_consensus.cc:482] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:47:15.896221 8439 raft_consensus.cc:504] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.896631 8439 leader_election.cc:288] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155), ada18d69250940839ecbf049c4181dca (127.0.98.3:43111)
I20260924 04:47:15.897007 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3426afd69f3a45e784e3e10721a1da57" candidate_uuid: "7e9c3ee360cd4a28ac4715a826332653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada18d69250940839ecbf049c4181dca" is_pre_election: true
I20260924 04:47:15.897023 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3426afd69f3a45e784e3e10721a1da57" candidate_uuid: "7e9c3ee360cd4a28ac4715a826332653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dfbe556b6874ef9ad97379390ccc222" is_pre_election: true
I20260924 04:47:15.897152 8358 raft_consensus.cc:2454] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c3ee360cd4a28ac4715a826332653 in term 0.
I20260924 04:47:15.897167 8218 raft_consensus.cc:2454] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e9c3ee360cd4a28ac4715a826332653 in term 0.
I20260924 04:47:15.897430 8012 leader_election.cc:302] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c3ee360cd4a28ac4715a826332653, ada18d69250940839ecbf049c4181dca; no voters:
I20260924 04:47:15.897610 8439 raft_consensus.cc:2781] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:47:15.897666 8439 raft_consensus.cc:482] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:47:15.897706 8439 raft_consensus.cc:3037] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:15.898645 8439 raft_consensus.cc:504] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.898945 8439 leader_election.cc:288] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [CANDIDATE]: Term 1 election: Requested vote from peers 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155), ada18d69250940839ecbf049c4181dca (127.0.98.3:43111)
I20260924 04:47:15.899302 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3426afd69f3a45e784e3e10721a1da57" candidate_uuid: "7e9c3ee360cd4a28ac4715a826332653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dfbe556b6874ef9ad97379390ccc222"
I20260924 04:47:15.899434 8218 raft_consensus.cc:3037] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:15.900266 8218 raft_consensus.cc:2454] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c3ee360cd4a28ac4715a826332653 in term 1.
I20260924 04:47:15.900506 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3426afd69f3a45e784e3e10721a1da57" candidate_uuid: "7e9c3ee360cd4a28ac4715a826332653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada18d69250940839ecbf049c4181dca"
I20260924 04:47:15.900535 8009 leader_election.cc:302] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dfbe556b6874ef9ad97379390ccc222, 7e9c3ee360cd4a28ac4715a826332653; no voters:
I20260924 04:47:15.900625 8358 raft_consensus.cc:3037] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:15.900697 8439 raft_consensus.cc:2781] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:15.900844 8439 raft_consensus.cc:686] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [term 1 LEADER]: Becoming Leader. State: Replica: 7e9c3ee360cd4a28ac4715a826332653, State: Running, Role: LEADER
I20260924 04:47:15.901005 8439 consensus_queue.cc:237] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.901685 8358 raft_consensus.cc:2454] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e9c3ee360cd4a28ac4715a826332653 in term 1.
I20260924 04:47:15.902868 7920 catalog_manager.cc:5495] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e9c3ee360cd4a28ac4715a826332653 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7e9c3ee360cd4a28ac4715a826332653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } health_report { overall_health: UNKNOWN } } }
I20260924 04:47:15.918177 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:15.914644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58152 (local address 127.0.98.1:33507)
0924 04:47:15.914857 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:15.914860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:15.914872 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:47:15.914915 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:15.914917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:15.914961 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:15.915096 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:47:15.915400 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.915956 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.917178 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:15.917420 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:15.917505 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:15.917751 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:47:15.917822 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:47:15.917878 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:15.917967 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 04:47:15.918015 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260924 04:47:15.920151 8358 raft_consensus.cc:1261] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Refusing update from remote peer 7e9c3ee360cd4a28ac4715a826332653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:15.920164 8218 raft_consensus.cc:1261] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Refusing update from remote peer 7e9c3ee360cd4a28ac4715a826332653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:15.920571 8439 consensus_queue.cc:1035] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:15.920851 8502 consensus_queue.cc:1035] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:15.929226 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:15.929385 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:15.931295 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6: Sending DeleteTablet for 3 replicas of tablet 3426afd69f3a45e784e3e10721a1da57
I20260924 04:47:15.931902 8058 tablet_service.cc:1514] Processing DeleteTablet for tablet 3426afd69f3a45e784e3e10721a1da57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38408
I20260924 04:47:15.931927 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet 3426afd69f3a45e784e3e10721a1da57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51442
I20260924 04:47:15.932088 8458 tablet_replica.cc:330] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653: stopping tablet replica
I20260924 04:47:15.932111 8198 tablet_service.cc:1514] Processing DeleteTablet for tablet 3426afd69f3a45e784e3e10721a1da57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34812
I20260924 04:47:15.932202 8458 raft_consensus.cc:2229] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:47:15.932303 8457 tablet_replica.cc:330] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca: stopping tablet replica
I20260924 04:47:15.932372 8458 raft_consensus.cc:2258] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:15.932415 8457 raft_consensus.cc:2229] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:15.932536 8457 raft_consensus.cc:2258] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:15.932696 8456 tablet_replica.cc:330] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222: stopping tablet replica
I20260924 04:47:15.932806 8456 raft_consensus.cc:2229] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:15.932922 8456 raft_consensus.cc:2258] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:15.933079 8458 ts_tablet_manager.cc:1905] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:15.933578 8456 ts_tablet_manager.cc:1905] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:15.935479 8458 ts_tablet_manager.cc:1918] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:15.935573 8458 log.cc:1196] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-0/wal/wals/3426afd69f3a45e784e3e10721a1da57
I20260924 04:47:15.935839 8458 ts_tablet_manager.cc:1939] T 3426afd69f3a45e784e3e10721a1da57 P 7e9c3ee360cd4a28ac4715a826332653: Deleting consensus metadata
I20260924 04:47:15.935899 8456 ts_tablet_manager.cc:1918] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:15.935837 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:47:15.935962 8456 log.cc:1196] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/3426afd69f3a45e784e3e10721a1da57
I20260924 04:47:15.936187 8456 ts_tablet_manager.cc:1939] T 3426afd69f3a45e784e3e10721a1da57 P 6dfbe556b6874ef9ad97379390ccc222: Deleting consensus metadata
W20260924 04:47:15.936244 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:47:15.936493 7905 catalog_manager.cc:4841] TS 7e9c3ee360cd4a28ac4715a826332653 (127.0.98.1:33507): tablet 3426afd69f3a45e784e3e10721a1da57 (table test-table [id=99d59b12ce1a43abad4e93c374316005]) successfully deleted
I20260924 04:47:15.937976 7905 catalog_manager.cc:4841] TS 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155): tablet 3426afd69f3a45e784e3e10721a1da57 (table test-table [id=99d59b12ce1a43abad4e93c374316005]) successfully deleted
I20260924 04:47:15.938256 8457 ts_tablet_manager.cc:1905] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:15.940768 8457 ts_tablet_manager.cc:1918] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:15.940842 8457 log.cc:1196] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/3426afd69f3a45e784e3e10721a1da57
I20260924 04:47:15.940852 8058 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0cb4e9a0b547ba84e58a3c24aac40a (DEFAULT_TABLE table=test-table [id=7ecba72a63a14f1d89cf18cdb1468246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:15.941051 8457 ts_tablet_manager.cc:1939] T 3426afd69f3a45e784e3e10721a1da57 P ada18d69250940839ecbf049c4181dca: Deleting consensus metadata
I20260924 04:47:15.941139 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0cb4e9a0b547ba84e58a3c24aac40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.942854 8433 tablet_bootstrap.cc:492] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653: Bootstrap starting.
I20260924 04:47:15.943305 7908 catalog_manager.cc:4841] TS ada18d69250940839ecbf049c4181dca (127.0.98.3:43111): tablet 3426afd69f3a45e784e3e10721a1da57 (table test-table [id=99d59b12ce1a43abad4e93c374316005]) successfully deleted
I20260924 04:47:15.943584 8337 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0cb4e9a0b547ba84e58a3c24aac40a (DEFAULT_TABLE table=test-table [id=7ecba72a63a14f1d89cf18cdb1468246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:15.943760 8433 tablet_bootstrap.cc:654] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.943804 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0cb4e9a0b547ba84e58a3c24aac40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.944767 8433 tablet_bootstrap.cc:492] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653: No bootstrap required, opened a new log
I20260924 04:47:15.944839 8433 ts_tablet_manager.cc:1397] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:47:15.945164 8433 raft_consensus.cc:348] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.945251 8433 raft_consensus.cc:374] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.945271 8433 raft_consensus.cc:729] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9c3ee360cd4a28ac4715a826332653, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.945423 8433 consensus_queue.cc:260] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.945672 8433 ts_tablet_manager.cc:1428] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:15.946189 8434 tablet_bootstrap.cc:492] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca: Bootstrap starting.
I20260924 04:47:15.946154 8198 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0cb4e9a0b547ba84e58a3c24aac40a (DEFAULT_TABLE table=test-table [id=7ecba72a63a14f1d89cf18cdb1468246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:15.946394 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0cb4e9a0b547ba84e58a3c24aac40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:15.946846 8434 tablet_bootstrap.cc:654] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.947783 8434 tablet_bootstrap.cc:492] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca: No bootstrap required, opened a new log
I20260924 04:47:15.947844 8434 ts_tablet_manager.cc:1397] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:47:15.948132 8434 raft_consensus.cc:348] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.948207 8434 raft_consensus.cc:374] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.948226 8434 raft_consensus.cc:729] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada18d69250940839ecbf049c4181dca, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.948326 8434 consensus_queue.cc:260] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.948583 8434 ts_tablet_manager.cc:1428] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:15.948621 8432 tablet_bootstrap.cc:492] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222: Bootstrap starting.
I20260924 04:47:15.949515 8432 tablet_bootstrap.cc:654] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:15.950644 8432 tablet_bootstrap.cc:492] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222: No bootstrap required, opened a new log
I20260924 04:47:15.950708 8432 ts_tablet_manager.cc:1397] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:47:15.951086 8432 raft_consensus.cc:348] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.951190 8432 raft_consensus.cc:374] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:15.951221 8432 raft_consensus.cc:729] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dfbe556b6874ef9ad97379390ccc222, State: Initialized, Role: FOLLOWER
I20260924 04:47:15.951333 8432 consensus_queue.cc:260] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:15.951586 8432 ts_tablet_manager.cc:1428] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:47:16.117226 8483 raft_consensus.cc:482] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:47:16.117367 8483 raft_consensus.cc:504] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:16.117817 8483 leader_election.cc:288] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e9c3ee360cd4a28ac4715a826332653 (127.0.98.1:33507), 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155)
I20260924 04:47:16.118297 8077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0cb4e9a0b547ba84e58a3c24aac40a" candidate_uuid: "ada18d69250940839ecbf049c4181dca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9c3ee360cd4a28ac4715a826332653" is_pre_election: true
I20260924 04:47:16.118316 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0cb4e9a0b547ba84e58a3c24aac40a" candidate_uuid: "ada18d69250940839ecbf049c4181dca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dfbe556b6874ef9ad97379390ccc222" is_pre_election: true
I20260924 04:47:16.118438 8218 raft_consensus.cc:2454] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada18d69250940839ecbf049c4181dca in term 0.
I20260924 04:47:16.118449 8077 raft_consensus.cc:2454] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada18d69250940839ecbf049c4181dca in term 0.
I20260924 04:47:16.118741 8289 leader_election.cc:302] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c3ee360cd4a28ac4715a826332653, ada18d69250940839ecbf049c4181dca; no voters:
I20260924 04:47:16.118903 8483 raft_consensus.cc:2781] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:47:16.118988 8483 raft_consensus.cc:482] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:47:16.119037 8483 raft_consensus.cc:3037] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:16.120045 8483 raft_consensus.cc:504] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:16.120338 8483 leader_election.cc:288] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [CANDIDATE]: Term 1 election: Requested vote from peers 7e9c3ee360cd4a28ac4715a826332653 (127.0.98.1:33507), 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155)
I20260924 04:47:16.120728 8077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0cb4e9a0b547ba84e58a3c24aac40a" candidate_uuid: "ada18d69250940839ecbf049c4181dca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9c3ee360cd4a28ac4715a826332653"
I20260924 04:47:16.120745 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0cb4e9a0b547ba84e58a3c24aac40a" candidate_uuid: "ada18d69250940839ecbf049c4181dca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dfbe556b6874ef9ad97379390ccc222"
I20260924 04:47:16.120854 8077 raft_consensus.cc:3037] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:16.120858 8218 raft_consensus.cc:3037] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:16.121727 8077 raft_consensus.cc:2454] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada18d69250940839ecbf049c4181dca in term 1.
I20260924 04:47:16.121748 8218 raft_consensus.cc:2454] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada18d69250940839ecbf049c4181dca in term 1.
I20260924 04:47:16.121987 8289 leader_election.cc:302] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9c3ee360cd4a28ac4715a826332653, ada18d69250940839ecbf049c4181dca; no voters:
I20260924 04:47:16.122151 8483 raft_consensus.cc:2781] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:16.122241 8483 raft_consensus.cc:686] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [term 1 LEADER]: Becoming Leader. State: Replica: ada18d69250940839ecbf049c4181dca, State: Running, Role: LEADER
I20260924 04:47:16.122376 8483 consensus_queue.cc:237] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } }
I20260924 04:47:16.123986 7919 catalog_manager.cc:5495] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca reported cstate change: term changed from 0 to 1, leader changed from <none> to ada18d69250940839ecbf049c4181dca (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ada18d69250940839ecbf049c4181dca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada18d69250940839ecbf049c4181dca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43111 } health_report { overall_health: HEALTHY } } }
I20260924 04:47:16.151350 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:16.148616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53220 (local address 127.0.98.3:43111)
0924 04:47:16.148712 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:16.148716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:16.148730 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:47:16.148839 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:16.148841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:16.148879 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:16.148936 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:47:16.149128 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:16.149620 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:16.150439 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:16.150659 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:16.150751 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:16.150957 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:47:16.151061 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:47:16.151115 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:16.151200 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 04:47:16.151243 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 04:47:16.157017 8218 raft_consensus.cc:1261] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Refusing update from remote peer ada18d69250940839ecbf049c4181dca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:16.157244 8077 raft_consensus.cc:1261] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Refusing update from remote peer ada18d69250940839ecbf049c4181dca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:16.157375 8463 consensus_queue.cc:1035] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dfbe556b6874ef9ad97379390ccc222" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:16.157697 8463 consensus_queue.cc:1035] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9c3ee360cd4a28ac4715a826332653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:16.186153 7919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:16.186311 7919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:16.188371 7919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a29aecc5b5f49888ea7d3858d2b12a6: Sending DeleteTablet for 3 replicas of tablet 2f0cb4e9a0b547ba84e58a3c24aac40a
I20260924 04:47:16.188930 8058 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0cb4e9a0b547ba84e58a3c24aac40a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38408
I20260924 04:47:16.188999 8198 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0cb4e9a0b547ba84e58a3c24aac40a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34812
I20260924 04:47:16.189127 8458 tablet_replica.cc:330] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653: stopping tablet replica
I20260924 04:47:16.189148 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0cb4e9a0b547ba84e58a3c24aac40a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51442
I20260924 04:47:16.189193 8456 tablet_replica.cc:330] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222: stopping tablet replica
I20260924 04:47:16.189244 8458 raft_consensus.cc:2229] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:16.189292 8457 tablet_replica.cc:330] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca: stopping tablet replica
I20260924 04:47:16.189294 8456 raft_consensus.cc:2229] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:16.189353 8458 raft_consensus.cc:2258] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:16.189378 8457 raft_consensus.cc:2229] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:47:16.189406 8456 raft_consensus.cc:2258] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:16.189556 8457 raft_consensus.cc:2258] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:16.190018 8456 ts_tablet_manager.cc:1905] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:16.190157 8458 ts_tablet_manager.cc:1905] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 7e9c3ee360cd4a28ac4715a826332653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:16.190796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 7993
I20260924 04:47:16.192487 8456 ts_tablet_manager.cc:1918] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:47:16.192565 8456 log.cc:1196] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/2f0cb4e9a0b547ba84e58a3c24aac40a
I20260924 04:47:16.192786 8456 ts_tablet_manager.cc:1939] T 2f0cb4e9a0b547ba84e58a3c24aac40a P 6dfbe556b6874ef9ad97379390ccc222: Deleting consensus metadata
I20260924 04:47:16.193534 8457 ts_tablet_manager.cc:1905] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:16.193598 7905 catalog_manager.cc:4841] TS 6dfbe556b6874ef9ad97379390ccc222 (127.0.98.2:42155): tablet 2f0cb4e9a0b547ba84e58a3c24aac40a (table test-table [id=7ecba72a63a14f1d89cf18cdb1468246]) successfully deleted
I20260924 04:47:16.195665 8457 ts_tablet_manager.cc:1918] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:47:16.195735 8457 log.cc:1196] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/2f0cb4e9a0b547ba84e58a3c24aac40a
I20260924 04:47:16.195926 8457 ts_tablet_manager.cc:1939] T 2f0cb4e9a0b547ba84e58a3c24aac40a P ada18d69250940839ecbf049c4181dca: Deleting consensus metadata
I20260924 04:47:16.196578 7908 catalog_manager.cc:4841] TS ada18d69250940839ecbf049c4181dca (127.0.98.3:43111): tablet 2f0cb4e9a0b547ba84e58a3c24aac40a (table test-table [id=7ecba72a63a14f1d89cf18cdb1468246]) successfully deleted
W20260924 04:47:16.198379 7905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33507 (error 108)
I20260924 04:47:16.198473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 8133
I20260924 04:47:16.200317 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:16.199979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33507 (local address 127.0.0.1:58162)
0924 04:47:16.200179 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:16.200237 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":59,"threads_started":1}
W20260924 04:47:16.200443 7905 catalog_manager.cc:4584] TS 7e9c3ee360cd4a28ac4715a826332653 (127.0.98.1:33507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f0cb4e9a0b547ba84e58a3c24aac40a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33507: connect: Connection refused (error 111)
I20260924 04:47:16.203900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 8273
I20260924 04:47:16.209625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 7889
2026-09-24T04:47:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3153 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:47:16 dist-test-slave-hvcl krb5kdc[8519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:47:16 dist-test-slave-hvcl krb5kdc[8519](info): set up 2 sockets
Sep 24 04:47:16 dist-test-slave-hvcl krb5kdc[8519](info): commencing operation
krb5kdc: starting...
W20260924 04:47:17.199337 392 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:47:17 dist-test-slave-hvcl krb5kdc[8519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:47:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:47:17.319119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32857
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:17.428418 8532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:17.428755 8532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:17.428803 8532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:17.432962 8532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:17.433037 8532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:17.433056 8532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:17.433072 8532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:17.433086 8532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:47:17.438627 8532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:17.439769 8532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:17.440984 8532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:47:17.449980 8532 server_base.cc:1034] running on GCE node
W20260924 04:47:17.449901 8537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:17.449918 8540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:17.449901 8538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:17.450785 8532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:17.451978 8532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:17.453152 8532 hybrid_clock.cc:648] HybridClock initialized: now 1790225237453136 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T04:47:17Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1247 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:47:17 dist-test-slave-hvcl krb5kdc[8545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:47:17 dist-test-slave-hvcl krb5kdc[8545](info): set up 2 sockets
Sep 24 04:47:17 dist-test-slave-hvcl krb5kdc[8545](info): commencing operation
krb5kdc: starting...
W20260924 04:47:18.411422 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:47:18 dist-test-slave-hvcl krb5kdc[8545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:47:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:47:18.530653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35463
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:18.637750 8558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:18.638010 8558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:18.638064 8558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:18.642056 8558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:18.642119 8558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:18.642146 8558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:18.642167 8558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:18.642189 8558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:47:18.647514 8558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:18.648523 8558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:18.649523 8558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:18.657765 8566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:18.657783 8564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:18.657784 8563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:18.658138 8558 server_base.cc:1034] running on GCE node
I20260924 04:47:18.658666 8558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:18.659762 8558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:18.660914 8558 hybrid_clock.cc:648] HybridClock initialized: now 1790225238660893 us; error 34 us; skew 500 ppm
Sep 24 04:47:18 dist-test-slave-hvcl krb5kdc[8545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:18.668581 8558 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:47:18.669564 8558 webserver.cc:460] Webserver started at http://127.0.98.62:45601/ using document root <none> and password file <none>
I20260924 04:47:18.670125 8558 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:18.670187 8558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:18.670398 8558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:18.672528 8558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "20303abdfc524ce6bb0667e9ca4179cd"
format_stamp: "Formatted at 2026-09-24 04:47:18 on dist-test-slave-hvcl"
server_key: "e42b7deb91af3d506e35f69a51f3423c"
server_key_iv: "e84e791c20a44c13f9b49002269816c2"
server_key_version: "encryptionkey@0"
I20260924 04:47:18.673082 8558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "20303abdfc524ce6bb0667e9ca4179cd"
format_stamp: "Formatted at 2026-09-24 04:47:18 on dist-test-slave-hvcl"
server_key: "e42b7deb91af3d506e35f69a51f3423c"
server_key_iv: "e84e791c20a44c13f9b49002269816c2"
server_key_version: "encryptionkey@0"
I20260924 04:47:18.676905 8558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:47:18.679462 8573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:18.680693 8558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:18.680794 8558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "20303abdfc524ce6bb0667e9ca4179cd"
format_stamp: "Formatted at 2026-09-24 04:47:18 on dist-test-slave-hvcl"
server_key: "e42b7deb91af3d506e35f69a51f3423c"
server_key_iv: "e84e791c20a44c13f9b49002269816c2"
server_key_version: "encryptionkey@0"
I20260924 04:47:18.680888 8558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:18.697317 8558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:18.700466 8558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:18.700639 8558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:18.707580 8558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35463
I20260924 04:47:18.707646 8625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35463 every 8 connection(s)
I20260924 04:47:18.708559 8558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:47:18.711371 8626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:18.716823 8626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd: Bootstrap starting.
I20260924 04:47:18.717144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 8558
I20260924 04:47:18.717280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:47:18.717684 392 external_mini_cluster.cc:1428] Setting key ce0157c1bb85177a441fdcb07bd96816
I20260924 04:47:18.719843 8626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:18.720603 8626 log.cc:824] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:18.722709 8626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd: No bootstrap required, opened a new log
I20260924 04:47:18.726619 8626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20303abdfc524ce6bb0667e9ca4179cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35463 } }
Sep 24 04:47:18 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:47:18.726873 8626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:18.726951 8626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20303abdfc524ce6bb0667e9ca4179cd, State: Initialized, Role: FOLLOWER
I20260924 04:47:18.727453 8626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20303abdfc524ce6bb0667e9ca4179cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35463 } }
I20260924 04:47:18.727638 8626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:47:18.727705 8626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:47:18.727820 8626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:18.728667 8626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20303abdfc524ce6bb0667e9ca4179cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35463 } }
I20260924 04:47:18.729000 8626 leader_election.cc:302] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20303abdfc524ce6bb0667e9ca4179cd; no voters:
I20260924 04:47:18.729278 8626 leader_election.cc:288] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:47:18.729404 8631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:18.729592 8631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 1 LEADER]: Becoming Leader. State: Replica: 20303abdfc524ce6bb0667e9ca4179cd, State: Running, Role: LEADER
I20260924 04:47:18.729933 8631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20303abdfc524ce6bb0667e9ca4179cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35463 } }
I20260924 04:47:18.730556 8626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:18.731477 8632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20303abdfc524ce6bb0667e9ca4179cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20303abdfc524ce6bb0667e9ca4179cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35463 } } }
I20260924 04:47:18.731602 8632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:18.731837 8633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20303abdfc524ce6bb0667e9ca4179cd. Latest consensus state: current_term: 1 leader_uuid: "20303abdfc524ce6bb0667e9ca4179cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20303abdfc524ce6bb0667e9ca4179cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35463 } } }
I20260924 04:47:18.731984 8633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:18.732736 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:18.719165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55958 (local address 127.0.98.62:35463)
0924 04:47:18.719712 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:18.719727 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:18.719763 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:47:18.720444 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:18.720461 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:18.720846 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:18.721141 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:18.721864 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:18.723326 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:18.724191 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:18.724871 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:18.725033 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:18.727374 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:18.727381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:18.727397 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:18.727427 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:18.729284 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:18.729811 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:18.729816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:18.729822 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:18.729951 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:18.730315 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:18.730318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:18.730320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:18.730818 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:18.731066 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:18.731312 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 04:47:18.731630 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":135,"threads_started":1}
W20260924 04:47:18.735850 8646 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:47:18.735915 8646 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:47:18.735996 8640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:18.736678 8640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:18.741789 8640 catalog_manager.cc:1347] Generated new cluster ID: 8d34b348b3e9478089e70f8f958dfa01
I20260924 04:47:18.741863 8640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:18.756511 8640 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:47:18.757475 8640 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:18.763432 8640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd: Generated new TSK 0
I20260924 04:47:18.764148 8640 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:47:18.798600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35463
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:18.938186 8652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:18.938450 8652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:18.938493 8652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:18.942541 8652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:18.942613 8652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:18.942674 8652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:47:18.948133 8652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:18.949249 8652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:18.950274 8652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:18.960258 8660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:18.960228 8657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:18.960229 8658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:18.960445 8652 server_base.cc:1034] running on GCE node
I20260924 04:47:18.960840 8652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:18.961460 8652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:18.962627 8652 hybrid_clock.cc:648] HybridClock initialized: now 1790225238962595 us; error 47 us; skew 500 ppm
Sep 24 04:47:18 dist-test-slave-hvcl krb5kdc[8545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:18.965440 8652 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:47:18.966323 8652 webserver.cc:460] Webserver started at http://127.0.98.1:45745/ using document root <none> and password file <none>
I20260924 04:47:18.966851 8652 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:18.966908 8652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:18.967121 8652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:18.969151 8652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "b33594bd87a742a7afd9e132ffa6d83b"
format_stamp: "Formatted at 2026-09-24 04:47:18 on dist-test-slave-hvcl"
server_key: "0c444e587934a8a537c256299aba924d"
server_key_iv: "704ffeb2bb9cfe1c287c590d13bc3e7e"
server_key_version: "encryptionkey@0"
I20260924 04:47:18.969671 8652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b33594bd87a742a7afd9e132ffa6d83b"
format_stamp: "Formatted at 2026-09-24 04:47:18 on dist-test-slave-hvcl"
server_key: "0c444e587934a8a537c256299aba924d"
server_key_iv: "704ffeb2bb9cfe1c287c590d13bc3e7e"
server_key_version: "encryptionkey@0"
I20260924 04:47:18.973486 8652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:47:18.976051 8667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:18.977344 8652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:47:18.977463 8652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "b33594bd87a742a7afd9e132ffa6d83b"
format_stamp: "Formatted at 2026-09-24 04:47:18 on dist-test-slave-hvcl"
server_key: "0c444e587934a8a537c256299aba924d"
server_key_iv: "704ffeb2bb9cfe1c287c590d13bc3e7e"
server_key_version: "encryptionkey@0"
I20260924 04:47:18.977581 8652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:19.015591 8652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:19.018774 8652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:19.018962 8652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:19.019564 8652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:19.020354 8652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:19.020396 8652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:19.020438 8652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:19.020466 8652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:19.028714 8652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41671
I20260924 04:47:19.028749 8780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41671 every 8 connection(s)
I20260924 04:47:19.029598 8652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:47:19.036099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 8652
I20260924 04:47:19.036219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:47:19.036495 392 external_mini_cluster.cc:1428] Setting key 266e6472531e828f1de87c03b090b867
Sep 24 04:47:19 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:47:19.042824 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:19.031417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36615 (local address 127.0.98.62:35463)
0924 04:47:19.031569 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:19.031572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:19.032462 (+ 890us) server_negotiation.cc:407] Connection header received
0924 04:47:19.033749 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:19.033752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:19.033798 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:19.033927 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:19.035104 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.035647 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.036865 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.037156 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.037252 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:19.039523 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:19.039526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:19.039529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:19.039554 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:19.040898 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:19.041386 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:19.041389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:19.041391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:19.041480 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:19.041778 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:19.041781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:19.041783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:19.041967 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:19.042051 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:19.042532 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 04:47:19.042659 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 04:47:19.043711 8783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:19.031862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:36615)
0924 04:47:19.032304 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:19.032345 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:19.033546 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:19.034111 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:19.034117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:19.034454 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:19.034920 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:19.034933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.035755 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.035758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:19.036752 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:19.036761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.037291 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.037294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:19.037450 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:19.037979 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:19.037995 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:19.039361 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:19.041005 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:19.041011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:19.041027 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:19.041304 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:19.041558 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:19.041561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:19.041563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:19.041697 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:19.042061 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:19.042067 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:19.042310 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 04:47:19.042473 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 04:47:19.042800 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:47:19.045033 8781 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35463
I20260924 04:47:19.045253 8781 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:19.045713 8781 heartbeater.cc:511] Master 127.0.98.62:35463 requested a full tablet report, sending...
I20260924 04:47:19.047164 8588 ts_manager.cc:194] Registered new tserver with Master: b33594bd87a742a7afd9e132ffa6d83b (127.0.98.1:41671)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:47:19.048424 8588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:47:19.065913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35463
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:19.173666 8786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:19.173956 8786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:19.174002 8786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:19.178323 8786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:19.178400 8786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:19.178479 8786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:47:19.184139 8786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:19.185195 8786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:19.186230 8786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:19.195922 8792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:19.195890 8791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:19.195911 8794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:19.196293 8786 server_base.cc:1034] running on GCE node
I20260924 04:47:19.196669 8786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:19.197254 8786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:19.198412 8786 hybrid_clock.cc:648] HybridClock initialized: now 1790225239198379 us; error 48 us; skew 500 ppm
Sep 24 04:47:19 dist-test-slave-hvcl krb5kdc[8545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:19.201064 8786 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:47:19.201907 8786 webserver.cc:460] Webserver started at http://127.0.98.2:46529/ using document root <none> and password file <none>
I20260924 04:47:19.202447 8786 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:19.202505 8786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:19.202708 8786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:19.204722 8786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "de0a582caed44c6b8d9b1c173746db20"
format_stamp: "Formatted at 2026-09-24 04:47:19 on dist-test-slave-hvcl"
server_key: "72950b83b8dd79378f34f022d484840b"
server_key_iv: "ff8e6121671f719f09bfc7ce141b8409"
server_key_version: "encryptionkey@0"
I20260924 04:47:19.205245 8786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "de0a582caed44c6b8d9b1c173746db20"
format_stamp: "Formatted at 2026-09-24 04:47:19 on dist-test-slave-hvcl"
server_key: "72950b83b8dd79378f34f022d484840b"
server_key_iv: "ff8e6121671f719f09bfc7ce141b8409"
server_key_version: "encryptionkey@0"
I20260924 04:47:19.208966 8786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 04:47:19.211465 8801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:19.212806 8786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:19.212927 8786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "de0a582caed44c6b8d9b1c173746db20"
format_stamp: "Formatted at 2026-09-24 04:47:19 on dist-test-slave-hvcl"
server_key: "72950b83b8dd79378f34f022d484840b"
server_key_iv: "ff8e6121671f719f09bfc7ce141b8409"
server_key_version: "encryptionkey@0"
I20260924 04:47:19.213035 8786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:19.234450 8786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:19.237609 8786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:19.237793 8786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:19.238391 8786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:19.239240 8786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:19.239291 8786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:19.239344 8786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:19.239375 8786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:19.247829 8786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37897
I20260924 04:47:19.247866 8914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37897 every 8 connection(s)
I20260924 04:47:19.248755 8786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:47:19.252106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 8786
I20260924 04:47:19.252205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:47:19.252454 392 external_mini_cluster.cc:1428] Setting key 58bf21a992f7531da51eda08feaeae21
Sep 24 04:47:19 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:47:19.263092 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:19.250484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40363 (local address 127.0.98.62:35463)
0924 04:47:19.250641 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:19.250646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:19.251546 (+ 900us) server_negotiation.cc:407] Connection header received
0924 04:47:19.252991 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:19.252995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:19.253043 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:19.253109 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:19.254378 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.255026 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.256160 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.256407 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.256496 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:19.259246 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:19.259249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:19.259251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:19.259276 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:19.260770 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:19.261354 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:19.261357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:19.261358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:19.261436 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:19.261813 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:19.261816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:19.261817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:19.262069 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:19.262146 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:19.262762 (+ 616us) server_negotiation.cc:299] Negotiation successful
0924 04:47:19.262895 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:47:19.264053 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:19.250870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:40363)
0924 04:47:19.251397 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:19.251442 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:19.252771 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:19.253301 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:19.253308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:19.253732 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:19.254246 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:19.254259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.255133 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.255137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:19.256047 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:19.256057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.256547 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.256551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:19.256740 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:19.257330 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:19.257359 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:19.259066 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:19.260900 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:19.260906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:19.260922 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:19.261252 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:19.261547 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:19.261549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:19.261551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:19.261708 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:19.262172 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:19.262177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:19.262495 (+ 318us) client_negotiation.cc:769] Sending connection context
0924 04:47:19.262692 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 04:47:19.263131 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":135,"threads_started":1}
I20260924 04:47:19.265439 8915 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35463
I20260924 04:47:19.265681 8915 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:19.266182 8915 heartbeater.cc:511] Master 127.0.98.62:35463 requested a full tablet report, sending...
I20260924 04:47:19.267227 8588 ts_manager.cc:194] Registered new tserver with Master: de0a582caed44c6b8d9b1c173746db20 (127.0.98.2:37897)
I20260924 04:47:19.267869 8588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:47:19.280267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35463
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:19.388290 8920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:19.388545 8920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:19.388586 8920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:19.392594 8920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:19.392647 8920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:19.392710 8920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:47:19.397913 8920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:19.398926 8920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:19.399923 8920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:19.409507 8925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:19.409541 8926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:19.409509 8928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:19.409925 8920 server_base.cc:1034] running on GCE node
I20260924 04:47:19.410322 8920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:19.410957 8920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:19.412156 8920 hybrid_clock.cc:648] HybridClock initialized: now 1790225239412126 us; error 43 us; skew 500 ppm
Sep 24 04:47:19 dist-test-slave-hvcl krb5kdc[8545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:19.414990 8920 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:47:19.415887 8920 webserver.cc:460] Webserver started at http://127.0.98.3:37667/ using document root <none> and password file <none>
I20260924 04:47:19.416402 8920 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:19.416457 8920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:19.416615 8920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:19.418702 8920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "00db2837fe0e4b388c4b02e6db171f15"
format_stamp: "Formatted at 2026-09-24 04:47:19 on dist-test-slave-hvcl"
server_key: "7f05d5ed0ab24b7b2b6458d140a47c87"
server_key_iv: "95c0fb654030a628a9060e94086dcc01"
server_key_version: "encryptionkey@0"
I20260924 04:47:19.419252 8920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "00db2837fe0e4b388c4b02e6db171f15"
format_stamp: "Formatted at 2026-09-24 04:47:19 on dist-test-slave-hvcl"
server_key: "7f05d5ed0ab24b7b2b6458d140a47c87"
server_key_iv: "95c0fb654030a628a9060e94086dcc01"
server_key_version: "encryptionkey@0"
I20260924 04:47:19.423029 8920 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:47:19.425642 8935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:19.426924 8920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:47:19.427096 8920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "00db2837fe0e4b388c4b02e6db171f15"
format_stamp: "Formatted at 2026-09-24 04:47:19 on dist-test-slave-hvcl"
server_key: "7f05d5ed0ab24b7b2b6458d140a47c87"
server_key_iv: "95c0fb654030a628a9060e94086dcc01"
server_key_version: "encryptionkey@0"
I20260924 04:47:19.427209 8920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:19.451270 8920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:19.454527 8920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:19.454702 8920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:19.455401 8920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:19.456207 8920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:19.456277 8920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:19.456339 8920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:19.456357 8920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:19.464375 8920 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41811
I20260924 04:47:19.464449 9048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41811 every 8 connection(s)
I20260924 04:47:19.465265 8920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:47:19.466619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 8920
I20260924 04:47:19.466739 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:47:19.467075 392 external_mini_cluster.cc:1428] Setting key 552fffc720986151014e72fb6a8e56ad
Sep 24 04:47:19 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:47:19.479827 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:19.467371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59615 (local address 127.0.98.62:35463)
0924 04:47:19.467513 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:19.467518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:19.468345 (+ 827us) server_negotiation.cc:407] Connection header received
0924 04:47:19.469788 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:19.469791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:19.469837 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:19.469906 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:19.471077 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.471640 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.472856 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.473110 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.473212 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:19.476194 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:19.476197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:19.476199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:19.476224 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:19.477607 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:19.478136 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:19.478139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:19.478140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:19.478206 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:19.478560 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:19.478563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:19.478564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:19.478770 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:19.478916 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:19.479542 (+ 626us) server_negotiation.cc:299] Negotiation successful
0924 04:47:19.479668 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:47:19.480937 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:19.467736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:59615)
0924 04:47:19.468198 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:19.468251 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:19.469606 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:19.470059 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:19.470067 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:19.470429 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:19.470913 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:19.470927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.471788 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.471791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:19.472733 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:19.472743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.473289 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.473294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:19.473524 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:19.474231 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:19.474254 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:19.476032 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:19.477715 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:19.477720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:19.477734 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:19.478059 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:19.478279 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:19.478282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:19.478283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:19.478444 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:19.478948 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:19.478957 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:19.479269 (+ 312us) client_negotiation.cc:769] Sending connection context
0924 04:47:19.479459 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 04:47:19.479841 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":120,"threads_started":1}
I20260924 04:47:19.482137 9049 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35463
I20260924 04:47:19.482437 9049 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:19.483013 9049 heartbeater.cc:511] Master 127.0.98.62:35463 requested a full tablet report, sending...
I20260924 04:47:19.484261 8588 ts_manager.cc:194] Registered new tserver with Master: 00db2837fe0e4b388c4b02e6db171f15 (127.0.98.3:41811)
I20260924 04:47:19.484784 8588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59615
I20260924 04:47:19.492786 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:47:19.567297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 8652
I20260924 04:47:19.572291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 8786
I20260924 04:47:19.576853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 8920
I20260924 04:47:19.581261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 8558
I20260924 04:47:19.586685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35463
--webserver_interface=127.0.98.62
--webserver_port=45601
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:19.695269 9058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:19.695537 9058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:19.695597 9058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:19.700167 9058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:19.700235 9058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:19.700256 9058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:19.700271 9058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:19.700286 9058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:47:19.705797 9058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:19.706830 9058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:19.707854 9058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:19.715930 9063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:19.716025 9064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:19.715956 9066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:19.716246 9058 server_base.cc:1034] running on GCE node
I20260924 04:47:19.716795 9058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:19.717914 9058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:19.719051 9058 hybrid_clock.cc:648] HybridClock initialized: now 1790225239719031 us; error 35 us; skew 500 ppm
Sep 24 04:47:19 dist-test-slave-hvcl krb5kdc[8545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:19.721931 9058 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:47:19.722852 9058 webserver.cc:460] Webserver started at http://127.0.98.62:45601/ using document root <none> and password file <none>
I20260924 04:47:19.723479 9058 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:19.723562 9058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:19.727605 9058 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:47:19.729665 9073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:19.730851 9058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:19.730959 9058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "20303abdfc524ce6bb0667e9ca4179cd"
format_stamp: "Formatted at 2026-09-24 04:47:18 on dist-test-slave-hvcl"
server_key: "e42b7deb91af3d506e35f69a51f3423c"
server_key_iv: "e84e791c20a44c13f9b49002269816c2"
server_key_version: "encryptionkey@0"
I20260924 04:47:19.731518 9058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:19.767187 9058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:19.770496 9058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:19.770696 9058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:19.777872 9058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35463
I20260924 04:47:19.777981 9125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35463 every 8 connection(s)
I20260924 04:47:19.778924 9058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:47:19.782855 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 9058
I20260924 04:47:19.783164 9126 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:47:19.783384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41671
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35463
--webserver_port=45745
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:47:19.784792 9126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd: Bootstrap starting.
I20260924 04:47:19.800654 9126 log.cc:824] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:19.807149 9126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:47:19.807471 9126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd: Bootstrap complete.
I20260924 04:47:19.811434 9126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20303abdfc524ce6bb0667e9ca4179cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35463 } }
I20260924 04:47:19.811847 9126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20303abdfc524ce6bb0667e9ca4179cd, State: Initialized, Role: FOLLOWER
I20260924 04:47:19.812335 9126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20303abdfc524ce6bb0667e9ca4179cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35463 } }
I20260924 04:47:19.812454 9126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:47:19.812521 9126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:47:19.812614 9126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:47:19.814438 9126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20303abdfc524ce6bb0667e9ca4179cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35463 } }
I20260924 04:47:19.814743 9126 leader_election.cc:302] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20303abdfc524ce6bb0667e9ca4179cd; no voters:
I20260924 04:47:19.815056 9126 leader_election.cc:288] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:47:19.815160 9130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:47:19.815340 9130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [term 2 LEADER]: Becoming Leader. State: Replica: 20303abdfc524ce6bb0667e9ca4179cd, State: Running, Role: LEADER
I20260924 04:47:19.815637 9130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20303abdfc524ce6bb0667e9ca4179cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35463 } }
I20260924 04:47:19.816268 9126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:19.816545 9132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20303abdfc524ce6bb0667e9ca4179cd. Latest consensus state: current_term: 2 leader_uuid: "20303abdfc524ce6bb0667e9ca4179cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20303abdfc524ce6bb0667e9ca4179cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35463 } } }
I20260924 04:47:19.816666 9132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:19.816866 9131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "20303abdfc524ce6bb0667e9ca4179cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20303abdfc524ce6bb0667e9ca4179cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35463 } } }
I20260924 04:47:19.816957 9131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:19.817231 9134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:19.820392 9134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:19.821033 9134 catalog_manager.cc:1259] Loaded cluster ID: 8d34b348b3e9478089e70f8f958dfa01
I20260924 04:47:19.821085 9134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:19.822750 9134 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:19.823380 9134 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 20303abdfc524ce6bb0667e9ca4179cd: Loaded TSK: 0
I20260924 04:47:19.824301 9134 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:47:19.902045 9128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:19.902324 9128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:19.902385 9128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:19.906479 9128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:19.906555 9128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:19.906635 9128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:47:19.912014 9128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45745
--tserver_master_addrs=127.0.98.62:35463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:19.913064 9128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:19.914008 9128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:19.924417 9155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:19.924422 9152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:19.924599 9128 server_base.cc:1034] running on GCE node
W20260924 04:47:19.924715 9153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:19.925110 9128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:19.925729 9128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:19.926926 9128 hybrid_clock.cc:648] HybridClock initialized: now 1790225239926903 us; error 30 us; skew 500 ppm
Sep 24 04:47:19 dist-test-slave-hvcl krb5kdc[8545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225239, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:19.929697 9128 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:47:19.930596 9128 webserver.cc:460] Webserver started at http://127.0.98.1:45745/ using document root <none> and password file <none>
I20260924 04:47:19.931164 9128 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:19.931236 9128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:19.935199 9128 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:47:19.937367 9162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:19.938627 9128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:19.938728 9128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "b33594bd87a742a7afd9e132ffa6d83b"
format_stamp: "Formatted at 2026-09-24 04:47:18 on dist-test-slave-hvcl"
server_key: "0c444e587934a8a537c256299aba924d"
server_key_iv: "704ffeb2bb9cfe1c287c590d13bc3e7e"
server_key_version: "encryptionkey@0"
I20260924 04:47:19.939301 9128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:19.961266 9128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:19.964527 9128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:19.964699 9128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:19.965368 9128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:19.966205 9128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:19.966254 9128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:19.966293 9128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:19.966308 9128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:19.975214 9128 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41671
I20260924 04:47:19.975284 9275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41671 every 8 connection(s)
I20260924 04:47:19.976191 9128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:47:19.979545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 9128
I20260924 04:47:19.980062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37897
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35463
--webserver_port=46529
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:47:19 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:19 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:19.989445 9279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:19.978471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:55777)
0924 04:47:19.978921 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:19.978963 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:19.980508 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:19.981694 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:19.981702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:19.982112 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:19.982587 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:19.982602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.983919 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.983922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:19.984781 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:19.984790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.985510 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.985515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:19.985688 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:19.986231 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:19.986247 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:19.988444 (+ 2197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
W20260924 04:47:19.989437 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:19.978356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55777 (local address 127.0.98.62:35463)
0924 04:47:19.978813 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:19.978834 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:19.979094 (+ 260us) server_negotiation.cc:407] Connection header received
0924 04:47:19.980949 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:19.980961 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:19.981278 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:19.981599 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:19.982750 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.983782 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.984895 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.985354 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.985508 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:19.988676 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55777: BlockingRecv error: recv got EOF from 127.0.98.1:55777 (error 108)
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":103,"threads_started":1}
W20260924 04:47:19.990098 9276 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35463 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35463: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:47:19 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:19 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:19.995201 9279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:19.990504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:40683)
0924 04:47:19.990658 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:19.990671 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:19.990744 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:19.990955 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:19.990958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:19.991129 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:19.991213 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:19.991218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.991893 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.991897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:19.992442 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:19.992449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.992853 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.992856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:19.992944 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:19.993361 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:19.993374 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:19.995070 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93}
W20260924 04:47:19.995321 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:19.990603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40683 (local address 127.0.98.62:35463)
0924 04:47:19.990724 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:19.990728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:19.990740 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:47:19.990818 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:19.990821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:19.990862 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:19.990932 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:19.991314 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.991800 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.992542 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.992760 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.992842 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:19.995212 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40683: BlockingRecv error: recv got EOF from 127.0.98.1:40683 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 04:47:19 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:19 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:20.001219 9279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:19.995897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:50997)
0924 04:47:19.996052 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:19.996065 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:19.996131 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:19.996326 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:19.996329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:19.996489 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:19.996576 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:19.996582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.997416 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.997419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:19.998266 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:19.998275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.998712 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.998715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:19.998788 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:19.999299 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:19.999312 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:20.001091 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260924 04:47:20.001317 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:19.996046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50997 (local address 127.0.98.62:35463)
0924 04:47:19.996145 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:19.996148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:19.996162 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:47:19.996194 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:19.996196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:19.996230 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:19.996298 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:19.996691 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.997314 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.998388 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:19.998607 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:19.998692 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:20.001219 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50997: BlockingRecv error: recv got EOF from 127.0.98.1:50997 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 04:47:20.001514 9276 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:47:20.093758 9280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:20.094022 9280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:20.094064 9280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:20.098225 9280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:20.098289 9280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:20.098352 9280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:47:20.103613 9280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46529
--tserver_master_addrs=127.0.98.62:35463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:20.104624 9280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:20.105643 9280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:20.114926 9286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:20.115017 9285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:20.115052 9288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:20.115142 9280 server_base.cc:1034] running on GCE node
I20260924 04:47:20.115625 9280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:20.116405 9280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:20.117570 9280 hybrid_clock.cc:648] HybridClock initialized: now 1790225240117513 us; error 68 us; skew 500 ppm
Sep 24 04:47:20 dist-test-slave-hvcl krb5kdc[8545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225240, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:20.120393 9280 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:47:20.121253 9280 webserver.cc:460] Webserver started at http://127.0.98.2:46529/ using document root <none> and password file <none>
I20260924 04:47:20.121774 9280 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:20.121855 9280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:20.125641 9280 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:20.127686 9295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:20.128932 9280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:20.129032 9280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "de0a582caed44c6b8d9b1c173746db20"
format_stamp: "Formatted at 2026-09-24 04:47:19 on dist-test-slave-hvcl"
server_key: "72950b83b8dd79378f34f022d484840b"
server_key_iv: "ff8e6121671f719f09bfc7ce141b8409"
server_key_version: "encryptionkey@0"
I20260924 04:47:20.129565 9280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:20.147526 9280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:20.150563 9280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:20.150730 9280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:20.151343 9280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:20.152138 9280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:20.152184 9280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:20.152236 9280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:20.152266 9280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:20.160575 9280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37897
I20260924 04:47:20.160616 9408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37897 every 8 connection(s)
I20260924 04:47:20.161536 9280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:47:20.166312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 9280
I20260924 04:47:20.166736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41811
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35463
--webserver_port=37667
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:47:20 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:20 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:47:20.173048 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:20.163225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41657 (local address 127.0.98.62:35463)
0924 04:47:20.163347 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:20.163351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:20.164246 (+ 895us) server_negotiation.cc:407] Connection header received
0924 04:47:20.165535 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:20.165538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:20.165623 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:20.165707 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:20.166926 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.167491 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.169079 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.169305 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.169402 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:20.172925 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41657: BlockingRecv error: recv got EOF from 127.0.98.2:41657 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:47:20.173681 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:20.163614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:41657)
0924 04:47:20.164041 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:20.164135 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:20.165321 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:20.165888 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:20.165895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:20.166253 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:20.166756 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:20.166770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.167619 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.167622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:20.168936 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:20.168948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.169459 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.169463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:20.169660 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:20.170291 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:20.170313 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:20.172781 (+ 2468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
W20260924 04:47:20.174140 9409 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35463 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35463: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:47:20 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:20 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:20.179996 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:20.174586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:53031)
0924 04:47:20.174726 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:20.174741 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:20.174820 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:20.175088 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:20.175091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:20.175227 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:20.175322 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:20.175327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.176325 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.176328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:20.176905 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:20.176913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.177460 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.177462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:20.177553 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:20.178182 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:20.178194 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:20.179892 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 04:47:20.180164 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:20.174696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53031 (local address 127.0.98.62:35463)
0924 04:47:20.174838 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:20.174842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:20.174857 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:47:20.174900 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:20.174903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:20.174949 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:20.175066 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:20.175431 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.176206 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.177009 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.177360 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.177455 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:20.180039 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53031: BlockingRecv error: recv got EOF from 127.0.98.2:53031 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 04:47:20 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:20 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:20.185750 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:20.180696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:32969)
0924 04:47:20.180861 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:20.180873 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:20.180939 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:20.181130 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:20.181133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:20.181286 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:20.181370 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:20.181375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.182105 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.182109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:20.182956 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:20.182990 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.183426 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.183430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:20.183506 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:20.183909 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:20.183922 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:20.185626 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260924 04:47:20.185853 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:20.180797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32969 (local address 127.0.98.62:35463)
0924 04:47:20.180956 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:20.180959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:20.180972 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:47:20.181004 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:20.181006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:20.181040 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:20.181104 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:20.181466 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.182002 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.183126 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.183334 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.183418 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:20.185755 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32969: BlockingRecv error: recv got EOF from 127.0.98.2:32969 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 04:47:20.186141 9409 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:47:20.277310 9412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:20.277623 9412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:20.277688 9412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:20.281729 9412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:20.281798 9412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:20.281880 9412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:47:20.287220 9412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37667
--tserver_master_addrs=127.0.98.62:35463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:20.288288 9412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:20.289305 9412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:20.298439 9418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:20.298439 9417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:20.298473 9420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:20.298650 9412 server_base.cc:1034] running on GCE node
I20260924 04:47:20.299220 9412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:20.299853 9412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:20.301005 9412 hybrid_clock.cc:648] HybridClock initialized: now 1790225240300988 us; error 32 us; skew 500 ppm
Sep 24 04:47:20 dist-test-slave-hvcl krb5kdc[8545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225240, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:20.303709 9412 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:47:20.304551 9412 webserver.cc:460] Webserver started at http://127.0.98.3:37667/ using document root <none> and password file <none>
I20260924 04:47:20.305063 9412 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:20.305130 9412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:20.309027 9412 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:47:20.311365 9427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:20.312726 9412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:47:20.312824 9412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "00db2837fe0e4b388c4b02e6db171f15"
format_stamp: "Formatted at 2026-09-24 04:47:19 on dist-test-slave-hvcl"
server_key: "7f05d5ed0ab24b7b2b6458d140a47c87"
server_key_iv: "95c0fb654030a628a9060e94086dcc01"
server_key_version: "encryptionkey@0"
I20260924 04:47:20.313405 9412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:20.345274 9412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:20.348642 9412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:20.348840 9412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:20.349474 9412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:20.350283 9412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:20.350327 9412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:20.350365 9412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:20.350392 9412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:20.359028 9412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41811
I20260924 04:47:20.359154 9540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41811 every 8 connection(s)
I20260924 04:47:20.360023 9412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:47:20.363694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 9412
Sep 24 04:47:20 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:20 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:47:20.370927 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:20.361726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51629 (local address 127.0.98.62:35463)
0924 04:47:20.361880 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:20.361885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:20.362708 (+ 823us) server_negotiation.cc:407] Connection header received
0924 04:47:20.364076 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:20.364080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:20.364135 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:20.364315 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:20.365585 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.366096 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.367252 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.367514 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.367614 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:20.370825 (+ 3211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51629: BlockingRecv error: recv got EOF from 127.0.98.3:51629 (error 108)
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":59}
I20260924 04:47:20.372093 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:20.362112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:51629)
0924 04:47:20.362560 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:20.362610 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:20.363868 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:20.364497 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:20.364506 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:20.364971 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:20.365445 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:20.365458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.366204 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.366207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:20.367144 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:20.367154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.367662 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.367666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:20.367828 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:20.368274 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:20.368289 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:20.370710 (+ 2421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":105,"threads_started":1}
W20260924 04:47:20.372648 9541 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35463 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35463: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 04:47:20.373059 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:20.364768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.0.98.62:35463)
0924 04:47:20.364979 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:20.364983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:20.364993 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:47:20.365018 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:20.365020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:20.365048 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:20.365119 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:20.365789 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.366284 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.367192 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.367463 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.367554 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:20.368880 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:20.368886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:20.368899 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:20.368934 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:20.370831 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:20.371348 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:20.371354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:20.371361 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:20.371504 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:20.371800 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:20.371803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:20.371804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:20.372232 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:20.372422 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:20.372699 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 04:47:20.372910 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":155,"thread_start_us":84,"threads_started":1}
Sep 24 04:47:20 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:20 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:20.378245 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:20.373056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:55471)
0924 04:47:20.373322 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:20.373334 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:20.373415 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:20.373611 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:20.373614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:20.373757 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:20.373861 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:20.373868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.374891 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.374894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:20.375676 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:20.375685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.376162 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.376165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:20.376233 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:20.376609 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:20.376620 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:20.378153 (+ 1533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205}
W20260924 04:47:20.378378 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:20.373138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55471 (local address 127.0.98.62:35463)
0924 04:47:20.373245 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:20.373249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:20.373395 (+ 146us) server_negotiation.cc:407] Connection header received
0924 04:47:20.373486 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:20.373488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:20.373523 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:20.373582 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:20.373998 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.374758 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.375795 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.376060 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.376146 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:20.378268 (+ 2122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55471: BlockingRecv error: recv got EOF from 127.0.98.3:55471 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 04:47:20 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:20 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:20.383198 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:20.378733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:42427)
0924 04:47:20.378829 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:20.378842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:20.378904 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:20.379183 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:20.379186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:20.379303 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:20.379371 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:20.379376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.380139 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.380142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:20.380665 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:20.380673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.381121 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.381123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:20.381178 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:20.381587 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:20.381598 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:20.383103 (+ 1505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 04:47:20.383399 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:20.378797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42427 (local address 127.0.98.62:35463)
0924 04:47:20.378925 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:20.378928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:20.378938 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:47:20.378965 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:20.379054 (+ 89us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:20.379092 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:20.379154 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:20.379489 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.380033 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.380769 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:20.381030 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:20.381109 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:20.383240 (+ 2131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42427: BlockingRecv error: recv got EOF from 127.0.98.3:42427 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 04:47:20.383430 9541 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 04:47:21 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:21 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:21.007938 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:21.002312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:36571)
0924 04:47:21.002570 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:21.002586 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:21.002733 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:21.003032 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:21.003035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:21.003241 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:21.003384 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:21.003393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:21.004334 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:21.004337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:21.004926 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:21.004934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:21.005406 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:21.005409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:21.005481 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:21.006069 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:21.006081 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:21.007812 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
W20260924 04:47:21.008121 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:21.002410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36571 (local address 127.0.98.62:35463)
0924 04:47:21.002677 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:21.002681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:21.002696 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:47:21.002828 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:21.002830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:21.002878 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:21.003077 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:21.003510 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:21.004232 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:21.005080 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:21.005304 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:21.005388 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:21.007976 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36571: BlockingRecv error: recv got EOF from 127.0.98.1:36571 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
Sep 24 04:47:21 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:21 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:21.192555 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:21.186846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:38747)
0924 04:47:21.187187 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:21.187203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:21.187326 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:21.187602 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:21.187605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:21.187811 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:21.187905 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:21.187911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:21.188656 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:21.188659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:21.189249 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:21.189265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:21.189735 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:21.189739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:21.189828 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:21.190395 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:21.190411 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:21.192412 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":81,"threads_started":1}
W20260924 04:47:21.192684 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:21.187012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38747 (local address 127.0.98.62:35463)
0924 04:47:21.187162 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:21.187166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:21.187283 (+ 117us) server_negotiation.cc:407] Connection header received
0924 04:47:21.187421 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:21.187424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:21.187469 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:21.187562 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:21.188030 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:21.188551 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:21.189370 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:21.189630 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:21.189721 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:21.192564 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38747: BlockingRecv error: recv got EOF from 127.0.98.2:38747 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 04:47:21 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:21 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:21.389550 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:21.384018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:43597)
0924 04:47:21.384280 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:21.384296 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:21.384423 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:21.384678 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:21.384681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:21.384875 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:21.384980 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:21.384986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:21.385716 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:21.385719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:21.386380 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:21.386389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:21.386830 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:21.386833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:21.386922 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:21.387424 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:21.387436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:21.389417 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
W20260924 04:47:21.389705 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:21.384136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43597 (local address 127.0.98.62:35463)
0924 04:47:21.384280 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:21.384285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:21.384371 (+ 86us) server_negotiation.cc:407] Connection header received
0924 04:47:21.384506 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:21.384509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:21.384559 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:21.384643 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:21.385090 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:21.385617 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:21.386495 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:21.386732 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:21.386817 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:21.389579 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43597: BlockingRecv error: recv got EOF from 127.0.98.3:43597 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 04:47:22 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:22 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:22.015045 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:22.008843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:37705)
0924 04:47:22.009144 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:22.009159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:22.009300 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:22.009565 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:22.009567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:22.009755 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:22.009851 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:22.009858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:22.010732 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:22.010735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:22.011460 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:22.011472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:22.012089 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:22.012092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:22.012169 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:22.012815 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:22.012830 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:22.014832 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":72,"threads_started":1}
W20260924 04:47:22.015285 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:22.008945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37705 (local address 127.0.98.62:35463)
0924 04:47:22.009296 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:22.009300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:22.009316 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:47:22.009378 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:22.009380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:22.009419 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:22.009500 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:22.010014 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:22.010588 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:22.011609 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:22.011936 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:22.012035 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:22.015150 (+ 3115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37705: BlockingRecv error: recv got EOF from 127.0.98.1:37705 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":122,"threads_started":1}
Sep 24 04:47:22 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:22 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:22.199110 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:22.193428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:46911)
0924 04:47:22.193749 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:22.193765 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:22.193895 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:22.194159 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:22.194161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:22.194355 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:22.194461 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:22.194466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:22.195285 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:22.195289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:22.195861 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:22.195870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:22.196284 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:22.196289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:22.196366 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:22.196868 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:22.196882 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:22.198909 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":89,"threads_started":1}
W20260924 04:47:22.199220 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:22.193562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46911 (local address 127.0.98.62:35463)
0924 04:47:22.193752 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:22.193756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:22.193843 (+ 87us) server_negotiation.cc:407] Connection header received
0924 04:47:22.193990 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:22.193994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:22.194048 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:22.194117 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:22.194575 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:22.195179 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:22.195970 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:22.196187 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:22.196271 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:22.199105 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46911: BlockingRecv error: recv got EOF from 127.0.98.2:46911 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 24 04:47:22 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:22 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:22.395875 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:22.390393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:50557)
0924 04:47:22.390668 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:22.390681 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:22.390768 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:22.391046 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:22.391049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:22.391232 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:22.391334 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:22.391341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:22.392090 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:22.392092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:22.392654 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:22.392662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:22.393086 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:22.393089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:22.393155 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:22.393724 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:22.393737 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:22.395732 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
W20260924 04:47:22.396026 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:22.390574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50557 (local address 127.0.98.62:35463)
0924 04:47:22.390718 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:22.390721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:22.390735 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:47:22.390851 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:22.390854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:22.390899 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:22.390992 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:22.391439 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:22.391958 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:22.392775 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:22.392988 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:22.393064 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:22.395912 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50557: BlockingRecv error: recv got EOF from 127.0.98.3:50557 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 04:47:23 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:23 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:23.021718 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:23.015856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:41979)
0924 04:47:23.016171 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:23.016188 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:23.016349 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:23.016599 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:23.016602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:23.016776 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:23.016869 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:23.016874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:23.017835 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:23.017838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:23.018540 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:23.018552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:23.019057 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:23.019060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:23.019126 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:23.019671 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:23.019684 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:23.021591 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":92,"threads_started":1}
W20260924 04:47:23.021905 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:23.015940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41979 (local address 127.0.98.62:35463)
0924 04:47:23.016200 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:23.016204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:23.016334 (+ 130us) server_negotiation.cc:407] Connection header received
0924 04:47:23.016440 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:23.016442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:23.016487 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:23.016567 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:23.017015 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:23.017713 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:23.018677 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:23.018915 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:23.019034 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:23.021769 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41979: BlockingRecv error: recv got EOF from 127.0.98.1:41979 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":101,"threads_started":1}
Sep 24 04:47:23 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:23 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:23.205451 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:23.199982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:59825)
0924 04:47:23.200303 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:23.200362 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:23.200470 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:23.200783 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:23.200787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:23.201008 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:23.201132 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:23.201140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:23.202030 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:23.202033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:23.202625 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:23.202634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:23.203085 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:23.203089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:23.203159 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:23.203631 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:23.203642 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:23.205300 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":75,"threads_started":1}
W20260924 04:47:23.205591 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:23.200110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59825 (local address 127.0.98.62:35463)
0924 04:47:23.200268 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:23.200281 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:23.200435 (+ 154us) server_negotiation.cc:407] Connection header received
0924 04:47:23.200561 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:23.200564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:23.200620 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:23.200697 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:23.201241 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:23.201913 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:23.202740 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:23.202994 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:23.203069 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:23.205484 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59825: BlockingRecv error: recv got EOF from 127.0.98.2:59825 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 04:47:23 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:23 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:23.401932 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:23.396771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:38653)
0924 04:47:23.397034 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:23.397047 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:23.397147 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:23.397427 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:23.397430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:23.397606 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:23.397705 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:23.397710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:23.398434 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:23.398437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:23.399068 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:23.399077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:23.399496 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:23.399499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:23.399557 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:23.400032 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:23.400044 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:23.401783 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
W20260924 04:47:23.402015 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:23.396886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38653 (local address 127.0.98.62:35463)
0924 04:47:23.397012 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:23.397015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:23.397110 (+ 95us) server_negotiation.cc:407] Connection header received
0924 04:47:23.397218 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:23.397220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:23.397264 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:23.397346 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:23.397807 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:23.398319 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:23.399174 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:23.399403 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:23.399482 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:23.401904 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38653: BlockingRecv error: recv got EOF from 127.0.98.3:38653 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 04:47:24 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:24 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:24.031701 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:24.022509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:59663)
0924 04:47:24.022778 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:24.022790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:24.022907 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:24.023206 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:24.023208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:24.023390 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:24.023485 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:24.023490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:24.024346 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:24.024350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:24.024920 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:24.024928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:24.025410 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:24.025413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:24.025523 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:24.029576 (+ 4053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:24.029589 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:24.031551 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
W20260924 04:47:24.031934 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:24.022617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59663 (local address 127.0.98.62:35463)
0924 04:47:24.022837 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:24.022841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:24.022854 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:47:24.023042 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:24.023045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:24.023103 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:24.023181 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:24.023612 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:24.024232 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:24.025055 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:24.025288 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:24.025377 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:24.031705 (+ 6328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59663: BlockingRecv error: recv got EOF from 127.0.98.1:59663 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":55,"threads_started":1}
Sep 24 04:47:24 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:24 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:24.211555 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:24.206317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:50939)
0924 04:47:24.206561 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:24.206576 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:24.206669 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:24.206889 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:24.206890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:24.207093 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:24.207188 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:24.207192 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:24.208013 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:24.208016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:24.208597 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:24.208605 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:24.209022 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:24.209025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:24.209087 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:24.209598 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:24.209610 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:24.211407 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
W20260924 04:47:24.211654 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:24.206433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50939 (local address 127.0.98.62:35463)
0924 04:47:24.206559 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:24.206562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:24.206632 (+ 70us) server_negotiation.cc:407] Connection header received
0924 04:47:24.206753 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:24.206755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:24.206797 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:24.206876 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:24.207370 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:24.207896 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:24.208706 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:24.208928 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:24.209006 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:24.211554 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50939: BlockingRecv error: recv got EOF from 127.0.98.2:50939 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 04:47:24 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:24 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:24.408103 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:24.402767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:51991)
0924 04:47:24.403061 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:24.403074 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:24.403172 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:24.403439 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:24.403441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:24.403600 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:24.403706 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:24.403712 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:24.404546 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:24.404549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:24.405154 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:24.405163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:24.405606 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:24.405609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:24.405672 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:24.406147 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:24.406159 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:24.407954 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":67,"threads_started":1}
W20260924 04:47:24.408191 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:24.402870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51991 (local address 127.0.98.62:35463)
0924 04:47:24.403016 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:24.403020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:24.403136 (+ 116us) server_negotiation.cc:407] Connection header received
0924 04:47:24.403256 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:24.403258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:24.403304 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:24.403389 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:24.403841 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:24.404438 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:24.405260 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:24.405500 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:24.405583 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:24.408091 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51991: BlockingRecv error: recv got EOF from 127.0.98.3:51991 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 04:47:25 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:25 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:25.037976 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:25.032525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:60469)
0924 04:47:25.032767 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:25.032780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:25.032875 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:25.033206 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:25.033209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:25.033411 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:25.033563 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:25.033570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:25.034397 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:25.034400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:25.035070 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:25.035081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:25.035513 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:25.035517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:25.035595 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:25.036091 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:25.036109 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:25.037809 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
W20260924 04:47:25.038116 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:25.032602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60469 (local address 127.0.98.62:35463)
0924 04:47:25.032850 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:25.032854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:25.032868 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:47:25.032999 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:25.033002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:25.033049 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:25.033179 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:25.033683 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:25.034282 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:25.035177 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:25.035407 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:25.035494 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:25.037983 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60469: BlockingRecv error: recv got EOF from 127.0.98.1:60469 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
Sep 24 04:47:25 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:25 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:25.218045 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:25.212388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:53029)
0924 04:47:25.212703 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:25.212716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:25.212817 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:25.213123 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:25.213126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:25.213302 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:25.213406 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:25.213413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:25.214183 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:25.214185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:25.214781 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:25.214792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:25.215347 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:25.215351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:25.215432 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:25.215972 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:25.215987 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:25.217909 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":90,"threads_started":1}
W20260924 04:47:25.218164 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:25.212602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53029 (local address 127.0.98.62:35463)
0924 04:47:25.212734 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:25.212738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:25.212770 (+ 32us) server_negotiation.cc:407] Connection header received
0924 04:47:25.212925 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:25.212928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:25.212973 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:25.213041 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:25.213548 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:25.214066 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:25.214892 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:25.215213 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:25.215331 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:25.218055 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53029: BlockingRecv error: recv got EOF from 127.0.98.2:53029 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 04:47:25 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:25 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:25.414250 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:25.408951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:52619)
0924 04:47:25.409215 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:25.409227 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:25.409319 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:25.409589 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:25.409591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:25.409765 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:25.409862 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:25.409867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:25.410633 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:25.410636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:25.411267 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:25.411276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:25.411817 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:25.411820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:25.411896 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:25.412395 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:25.412406 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:25.414108 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
W20260924 04:47:25.414496 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:25.409053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52619 (local address 127.0.98.62:35463)
0924 04:47:25.409207 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:25.409211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:25.409290 (+ 79us) server_negotiation.cc:407] Connection header received
0924 04:47:25.409405 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:25.409407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:25.409460 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:25.409532 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:25.409989 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:25.410523 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:25.411465 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:25.411703 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:25.411789 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:25.414372 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52619: BlockingRecv error: recv got EOF from 127.0.98.3:52619 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 04:47:26 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:26 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:26.044535 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:26.038858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:45983)
0924 04:47:26.039157 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:26.039171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:26.039267 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:26.039526 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:26.039530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:26.039708 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:26.039805 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:26.039811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:26.040528 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:26.040531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:26.041109 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:26.041117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:26.041568 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:26.041571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:26.041655 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:26.042178 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:26.042190 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:26.044403 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":69,"threads_started":1}
W20260924 04:47:26.044665 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:26.038916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45983 (local address 127.0.98.62:35463)
0924 04:47:26.039215 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:26.039219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:26.039233 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:47:26.039356 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:26.039358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:26.039409 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:26.039501 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:26.039894 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:26.040411 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:26.041211 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:26.041433 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:26.041560 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:26.044549 (+ 2989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45983: BlockingRecv error: recv got EOF from 127.0.98.1:45983 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
Sep 24 04:47:26 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:26 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:26.223918 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:26.218753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:60441)
0924 04:47:26.219034 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:26.219049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:26.219141 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:26.219355 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:26.219357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:26.219582 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:26.219681 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:26.219687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:26.220428 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:26.220430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:26.221023 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:26.221031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:26.221479 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:26.221481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:26.221584 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:26.222048 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:26.222059 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:26.223798 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
W20260924 04:47:26.224025 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:26.218837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60441 (local address 127.0.98.62:35463)
0924 04:47:26.219016 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:26.219020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:26.219109 (+ 89us) server_negotiation.cc:407] Connection header received
0924 04:47:26.219209 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:26.219211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:26.219255 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:26.219333 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:26.219794 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:26.220344 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:26.221155 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:26.221383 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:26.221462 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:26.223926 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60441: BlockingRecv error: recv got EOF from 127.0.98.2:60441 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 04:47:26 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:26 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:26.421324 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:26.415132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:58915)
0924 04:47:26.415466 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:26.415480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:26.415619 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:26.415874 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:26.415877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:26.416099 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:26.416198 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:26.416204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:26.417174 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:26.417177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:26.417944 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:26.417957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:26.418585 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:26.418589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:26.418668 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:26.419175 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:26.419186 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:26.421200 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":101,"threads_started":1}
W20260924 04:47:26.421510 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:26.415238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58915 (local address 127.0.98.62:35463)
0924 04:47:26.415400 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:26.415405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:26.415640 (+ 235us) server_negotiation.cc:407] Connection header received
0924 04:47:26.415724 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:26.415727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:26.415775 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:26.415839 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:26.416352 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:26.417072 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:26.418089 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:26.418407 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:26.418525 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:26.421375 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58915: BlockingRecv error: recv got EOF from 127.0.98.3:58915 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 04:47:27 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:27 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:27.051076 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:27.045352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:49787)
0924 04:47:27.045746 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:27.045762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:27.045881 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:27.046251 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:27.046254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:27.046469 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:27.046589 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:27.046597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:27.047536 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:27.047539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:27.048108 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:27.048116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:27.048560 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:27.048564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:27.048633 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:27.049098 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:27.049109 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:27.050900 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":70,"threads_started":1}
W20260924 04:47:27.051199 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:27.045458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49787 (local address 127.0.98.62:35463)
0924 04:47:27.045731 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:27.045735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:27.045836 (+ 101us) server_negotiation.cc:407] Connection header received
0924 04:47:27.045996 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:27.045998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:27.046056 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:27.046158 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:27.046707 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:27.047441 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:27.048230 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:27.048461 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:27.048547 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:27.051084 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49787: BlockingRecv error: recv got EOF from 127.0.98.1:49787 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
Sep 24 04:47:27 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:27 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:27.229830 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:27.224711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:33347)
0924 04:47:27.225001 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:27.225015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:27.225133 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:27.225411 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:27.225414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:27.225600 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:27.225728 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:27.225734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:27.226484 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:27.226488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:27.227114 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:27.227123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:27.227518 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:27.227521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:27.227587 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:27.228064 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:27.228075 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:27.229716 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":75,"threads_started":1}
W20260924 04:47:27.229947 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:27.224814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33347 (local address 127.0.98.62:35463)
0924 04:47:27.224938 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:27.224942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:27.225088 (+ 146us) server_negotiation.cc:407] Connection header received
0924 04:47:27.225224 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:27.225227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:27.225281 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:27.225380 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:27.225833 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:27.226388 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:27.227205 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:27.227419 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:27.227494 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:27.229847 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33347: BlockingRecv error: recv got EOF from 127.0.98.2:33347 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 04:47:27 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:27 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:27.427443 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:27.422064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:33771)
0924 04:47:27.422397 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:27.422410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:27.422585 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:27.422850 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:27.422853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:27.423071 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:27.423184 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:27.423191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:27.423939 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:27.423943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:27.424508 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:27.424524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:27.424942 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:27.424945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:27.425023 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:27.425504 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:27.425523 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:27.427286 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":85,"threads_started":1}
W20260924 04:47:27.427536 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:27.422179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33771 (local address 127.0.98.62:35463)
0924 04:47:27.422348 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:27.422352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:27.422486 (+ 134us) server_negotiation.cc:407] Connection header received
0924 04:47:27.422692 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:27.422696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:27.422741 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:27.422808 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:27.423308 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:27.423834 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:27.424621 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:27.424840 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:27.424927 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:27.427433 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33771: BlockingRecv error: recv got EOF from 127.0.98.3:33771 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 04:47:28 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:28 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:28.057850 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:28.051868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:40595)
0924 04:47:28.052113 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:28.052126 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:28.052213 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:28.052500 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:28.052502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:28.052701 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:28.052798 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:28.052804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:28.053615 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:28.053618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:28.054287 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:28.054298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:28.054836 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:28.054840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:28.054926 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:28.055685 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:28.055700 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:28.057729 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":80,"threads_started":1}
W20260924 04:47:28.058044 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:28.052045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40595 (local address 127.0.98.62:35463)
0924 04:47:28.052281 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:28.052285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:28.052297 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:47:28.052334 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:28.052337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:28.052375 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:28.052457 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:28.052934 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:28.053489 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:28.054419 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:28.054709 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:28.054806 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:28.057886 (+ 3080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40595: BlockingRecv error: recv got EOF from 127.0.98.1:40595 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
Sep 24 04:47:28 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:28 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:28.236483 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:28.230727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:47681)
0924 04:47:28.231022 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:28.231036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:28.231151 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:28.231391 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:28.231394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:28.231617 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:28.231743 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:28.231751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:28.232681 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:28.232685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:28.233359 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:28.233368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:28.233865 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:28.233869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:28.233949 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:28.234432 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:28.234443 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:28.236334 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
W20260924 04:47:28.236569 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:28.230818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47681 (local address 127.0.98.62:35463)
0924 04:47:28.230957 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:28.230961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:28.231112 (+ 151us) server_negotiation.cc:407] Connection header received
0924 04:47:28.231242 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:28.231245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:28.231292 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:28.231359 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:28.231846 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:28.232575 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:28.233464 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:28.233759 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:28.233843 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:28.236470 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47681: BlockingRecv error: recv got EOF from 127.0.98.2:47681 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 04:47:28 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:28 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:28.433648 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:28.428257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:55427)
0924 04:47:28.428537 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:28.428552 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:28.428670 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:28.428931 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:28.428934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:28.429115 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:28.429224 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:28.429229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:28.430015 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:28.430018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:28.430566 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:28.430574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:28.431055 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:28.431059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:28.431148 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:28.431671 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:28.431686 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:28.433532 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
W20260924 04:47:28.433796 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:28.428369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55427 (local address 127.0.98.62:35463)
0924 04:47:28.428530 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:28.428534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:28.428618 (+ 84us) server_negotiation.cc:407] Connection header received
0924 04:47:28.428759 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:28.428762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:28.428807 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:28.428871 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:28.429338 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:28.429908 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:28.430677 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:28.430896 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:28.431017 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:28.433681 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55427: BlockingRecv error: recv got EOF from 127.0.98.3:55427 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 04:47:29 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:29 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:29.067153 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:29.058648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:59363)
0924 04:47:29.058924 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:29.058937 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:29.059066 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:29.059313 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:29.059315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:29.059487 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:29.059582 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:29.059587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:29.060481 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:29.060484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:29.061108 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:29.061118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:29.061573 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:29.061577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:29.061648 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:29.064933 (+ 3285us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:29.064949 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:29.066965 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
W20260924 04:47:29.067258 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:29.058789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59363 (local address 127.0.98.62:35463)
0924 04:47:29.059044 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:29.059047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:29.059060 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:47:29.059153 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:29.059155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:29.059199 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:29.059266 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:29.059790 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:29.060340 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:29.061221 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:29.061448 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:29.061531 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:29.067141 (+ 5610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59363: BlockingRecv error: recv got EOF from 127.0.98.1:59363 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
Sep 24 04:47:29 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:29 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:29.242847 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:29.237305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:49797)
0924 04:47:29.237563 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:29.237576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:29.237676 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:29.237947 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:29.237949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:29.238120 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:29.238217 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:29.238237 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:29.239086 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:29.239088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:29.239672 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:29.239681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:29.240138 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:29.240141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:29.240237 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:29.240723 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:29.240737 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:29.242684 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
W20260924 04:47:29.243019 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:29.237408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49797 (local address 127.0.98.62:35463)
0924 04:47:29.237554 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:29.237558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:29.237671 (+ 113us) server_negotiation.cc:407] Connection header received
0924 04:47:29.237785 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:29.237788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:29.237836 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:29.237910 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:29.238357 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:29.238932 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:29.239794 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:29.240017 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:29.240097 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:29.242878 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49797: BlockingRecv error: recv got EOF from 127.0.98.2:49797 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 04:47:29 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:29 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:29.439814 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:29.434443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:41025)
0924 04:47:29.434691 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:29.434706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:29.434819 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:29.435038 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:29.435040 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:29.435210 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:29.435320 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:29.435325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:29.436144 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:29.436148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:29.436716 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:29.436725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:29.437172 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:29.437176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:29.437256 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:29.437792 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:29.437804 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:29.439675 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
W20260924 04:47:29.439963 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:29.434491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41025 (local address 127.0.98.62:35463)
0924 04:47:29.434607 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:29.434610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:29.434781 (+ 171us) server_negotiation.cc:407] Connection header received
0924 04:47:29.434881 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:29.434883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:29.434925 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:29.435028 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:29.435471 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:29.436027 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:29.436831 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:29.437070 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:29.437157 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:29.439837 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41025: BlockingRecv error: recv got EOF from 127.0.98.3:41025 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 04:47:30 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:30 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:30.073688 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:30.068098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:60279)
0924 04:47:30.068364 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:30.068380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:30.068486 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:30.068827 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:30.068830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:30.069035 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:30.069156 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:30.069164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:30.069984 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:30.069987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:30.070589 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:30.070597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:30.071088 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:30.071092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:30.071166 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:30.071680 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:30.071691 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:30.073573 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
W20260924 04:47:30.073858 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:30.068136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60279 (local address 127.0.98.62:35463)
0924 04:47:30.068390 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:30.068395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:30.068440 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:47:30.068575 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:30.068577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:30.068689 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:30.068794 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:30.069302 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:30.069876 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:30.070697 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:30.070944 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:30.071049 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:30.073717 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60279: BlockingRecv error: recv got EOF from 127.0.98.1:60279 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
Sep 24 04:47:30 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:30 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:30.249046 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:30.243634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:41815)
0924 04:47:30.244026 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:30.244038 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:30.244118 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:30.244358 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:30.244362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:30.244576 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:30.244680 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:30.244686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:30.245450 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:30.245452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:30.246063 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:30.246072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:30.246488 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:30.246490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:30.246548 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:30.247055 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:30.247068 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:30.248922 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":75,"threads_started":1}
W20260924 04:47:30.249166 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:30.243727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41815 (local address 127.0.98.62:35463)
0924 04:47:30.243932 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:30.244114 (+ 182us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:30.244132 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:47:30.244179 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:30.244182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:30.244237 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:30.244309 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:30.244771 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:30.245335 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:30.246171 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:30.246402 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:30.246480 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:30.249053 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41815: BlockingRecv error: recv got EOF from 127.0.98.2:41815 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 04:47:30 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:30 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:30.446302 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:30.440652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:54425)
0924 04:47:30.440923 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:30.440938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:30.441037 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:30.441405 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:30.441408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:30.441590 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:30.441686 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:30.441691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:30.442481 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:30.442485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:30.443137 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:30.443148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:30.443635 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:30.443638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:30.443717 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:30.444262 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:30.444276 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:30.446184 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
W20260924 04:47:30.446427 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:30.440850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54425 (local address 127.0.98.62:35463)
0924 04:47:30.440973 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:30.440976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:30.440990 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:47:30.441157 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:30.441160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:30.441212 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:30.441326 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:30.441800 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:30.442375 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:30.443278 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:30.443538 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:30.443621 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:30.446324 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54425: BlockingRecv error: recv got EOF from 127.0.98.3:54425 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 04:47:31 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:31 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:31.079691 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:31.074461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:46509)
0924 04:47:31.074713 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:31.074727 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:31.074815 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:31.075150 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:31.075153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:31.075323 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:31.075448 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:31.075454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:31.076274 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:31.076277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:31.076838 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:31.076846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:31.077248 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:31.077250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:31.077307 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:31.077794 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:31.077806 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:31.079537 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
W20260924 04:47:31.079879 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:31.074632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46509 (local address 127.0.98.62:35463)
0924 04:47:31.074920 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:31.074923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:31.074937 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:47:31.074998 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:31.075000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:31.075041 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:31.075121 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:31.075566 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:31.076170 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:31.076941 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:31.077166 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:31.077241 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:31.079689 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46509: BlockingRecv error: recv got EOF from 127.0.98.1:46509 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":88,"threads_started":1}
Sep 24 04:47:31 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:31 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:31.255095 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:31.249749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:40037)
0924 04:47:31.250096 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:31.250109 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:31.250214 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:31.250448 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:31.250451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:31.250638 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:31.250727 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:31.250733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:31.251561 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:31.251565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:31.252105 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:31.252113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:31.252525 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:31.252529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:31.252608 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:31.253094 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:31.253108 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:31.254897 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":145,"threads_started":1}
W20260924 04:47:31.255174 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:31.249885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40037 (local address 127.0.98.62:35463)
0924 04:47:31.250029 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:31.250034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:31.250173 (+ 139us) server_negotiation.cc:407] Connection header received
0924 04:47:31.250298 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:31.250301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:31.250344 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:31.250406 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:31.250836 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:31.251458 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:31.252213 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:31.252429 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:31.252510 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:31.255071 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40037: BlockingRecv error: recv got EOF from 127.0.98.2:40037 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 04:47:31 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:31 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:31.452756 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:31.447232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:53637)
0924 04:47:31.447510 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:31.447525 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:31.447650 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:31.447913 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:31.447916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:31.448145 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:31.448243 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:31.448250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:31.449147 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:31.449149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:31.449714 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:31.449722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:31.450186 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:31.450190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:31.450254 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:31.450711 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:31.450723 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:31.452632 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":98,"threads_started":1}
W20260924 04:47:31.452960 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:31.447379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53637 (local address 127.0.98.62:35463)
0924 04:47:31.447521 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:31.447526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:31.447615 (+ 89us) server_negotiation.cc:407] Connection header received
0924 04:47:31.447736 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:31.447739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:31.447796 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:31.447865 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:31.448356 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:31.449043 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:31.449822 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:31.450079 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:31.450157 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:31.452779 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53637: BlockingRecv error: recv got EOF from 127.0.98.3:53637 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 04:47:32 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:32 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:32.085547 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:32.080432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:47637)
0924 04:47:32.080696 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:32.080712 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:32.080810 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:32.081030 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:32.081032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:32.081202 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:32.081299 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:32.081304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:32.082040 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:32.082042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:32.082615 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:32.082623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:32.083058 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:32.083060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:32.083121 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:32.083590 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:32.083602 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:32.085368 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":67,"threads_started":1}
W20260924 04:47:32.085649 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:32.080541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47637 (local address 127.0.98.62:35463)
0924 04:47:32.080798 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:32.080801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:32.080817 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:47:32.080876 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:32.080879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:32.080929 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:32.081007 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:32.081418 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:32.081958 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:32.082717 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:32.082955 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:32.083052 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:32.085535 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47637: BlockingRecv error: recv got EOF from 127.0.98.1:47637 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
Sep 24 04:47:32 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:32 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:32.261137 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:32.255899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:56047)
0924 04:47:32.256178 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:32.256194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:32.256314 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:32.256572 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:32.256575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:32.256760 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:32.256852 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:32.256857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:32.257587 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:32.257590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:32.258169 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:32.258177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:32.258591 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:32.258594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:32.258667 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:32.259190 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:32.259200 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:32.260975 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
W20260924 04:47:32.261202 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:32.256028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56047 (local address 127.0.98.62:35463)
0924 04:47:32.256181 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:32.256186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:32.256267 (+ 81us) server_negotiation.cc:407] Connection header received
0924 04:47:32.256401 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:32.256404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:32.256449 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:32.256516 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:32.256963 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:32.257489 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:32.258281 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:32.258495 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:32.258577 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:32.261108 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56047: BlockingRecv error: recv got EOF from 127.0.98.2:56047 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 04:47:32 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:32 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:32.458848 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:32.453477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:37293)
0924 04:47:32.453739 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:32.453754 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:32.453865 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:32.454164 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:32.454167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:32.454358 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:32.454455 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:32.454461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:32.455232 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:32.455235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:32.455793 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:32.455801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:32.456225 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:32.456228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:32.456305 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:32.456840 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:32.456853 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:32.458697 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
W20260924 04:47:32.459002 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:32.453579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37293 (local address 127.0.98.62:35463)
0924 04:47:32.453714 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:32.453718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:32.453823 (+ 105us) server_negotiation.cc:407] Connection header received
0924 04:47:32.453998 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:32.454001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:32.454046 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:32.454111 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:32.454565 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:32.455130 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:32.455903 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:32.456127 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:32.456211 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:32.458841 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37293: BlockingRecv error: recv got EOF from 127.0.98.3:37293 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 04:47:33 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:33 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:33.091511 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:33.086227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:45755)
0924 04:47:33.086490 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:33.086503 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:33.086597 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:33.086852 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:33.086854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:33.087120 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:33.087221 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:33.087227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:33.088006 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:33.088009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:33.088581 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:33.088589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:33.089020 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:33.089024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:33.089095 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:33.089570 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:33.089581 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:33.091387 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
W20260924 04:47:33.091661 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:33.086344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45755 (local address 127.0.98.62:35463)
0924 04:47:33.086546 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:33.086549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:33.086562 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:47:33.086673 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:33.086675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:33.086717 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:33.086789 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:33.087321 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:33.087886 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:33.088682 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:33.088899 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:33.088977 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:33.091516 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45755: BlockingRecv error: recv got EOF from 127.0.98.1:45755 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
Sep 24 04:47:33 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:33 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:33.267431 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:33.261983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:58879)
0924 04:47:33.262285 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:33.262300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:33.262406 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:33.262664 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:33.262666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:33.262855 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:33.263042 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:33.263051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:33.263925 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:33.263928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:33.264517 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:33.264525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:33.264926 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:33.264928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:33.265048 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:33.265515 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:33.265526 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:33.267312 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":101,"threads_started":1}
W20260924 04:47:33.267532 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:33.262215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58879 (local address 127.0.98.62:35463)
0924 04:47:33.262353 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:33.262358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:33.262375 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:47:33.262499 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:33.262502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:33.262545 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:33.262616 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:33.263154 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:33.263810 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:33.264623 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:33.264830 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:33.264897 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:33.267437 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58879: BlockingRecv error: recv got EOF from 127.0.98.2:58879 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 04:47:33 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:33 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:33.465130 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:33.459730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:46389)
0924 04:47:33.459970 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:33.459983 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:33.460093 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:33.460362 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:33.460365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:33.460571 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:33.460667 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:33.460690 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:33.461436 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:33.461439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:33.461995 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:33.462003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:33.462531 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:33.462535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:33.462627 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:33.463222 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:33.463234 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:33.464963 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":86,"threads_started":1}
W20260924 04:47:33.465205 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:33.459769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46389 (local address 127.0.98.62:35463)
0924 04:47:33.459898 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:33.459902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:33.460054 (+ 152us) server_negotiation.cc:407] Connection header received
0924 04:47:33.460207 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:33.460210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:33.460260 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:33.460327 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:33.460806 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:33.461333 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:33.462107 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:33.462372 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:33.462473 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:33.465096 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46389: BlockingRecv error: recv got EOF from 127.0.98.3:46389 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 04:47:34 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:34 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:34.103479 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:34.092355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:46121)
0924 04:47:34.092620 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:34.092637 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:34.092748 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:34.093017 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:34.093019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:34.093194 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:34.093290 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:34.093295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:34.094080 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:34.094083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:34.094658 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:34.094668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:34.095135 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:34.095138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:34.095219 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:34.101327 (+ 6108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:34.101341 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:34.103339 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
W20260924 04:47:34.103658 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:34.092462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46121 (local address 127.0.98.62:35463)
0924 04:47:34.092799 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:34.092803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:34.092817 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:47:34.092858 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:34.092860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:34.092900 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:34.092986 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:34.093418 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:34.093965 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:34.094753 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:34.095007 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:34.095090 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:34.103505 (+ 8415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46121: BlockingRecv error: recv got EOF from 127.0.98.1:46121 (error 108)
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":96,"threads_started":1}
Sep 24 04:47:34 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:34 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:34.273684 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:34.268170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:59093)
0924 04:47:34.268488 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:34.268503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:34.268637 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:34.268921 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:34.268925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:34.269119 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:34.269213 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:34.269218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:34.269972 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:34.269976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:34.270578 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:34.270587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:34.271061 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:34.271065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:34.271144 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:34.271712 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:34.271724 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:34.273512 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":90,"threads_started":1}
W20260924 04:47:34.273774 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:34.268319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59093 (local address 127.0.98.62:35463)
0924 04:47:34.268489 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:34.268494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:34.268585 (+ 91us) server_negotiation.cc:407] Connection header received
0924 04:47:34.268725 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:34.268750 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:34.268797 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:34.268877 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:34.269342 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:34.269865 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:34.270703 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:34.270930 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:34.271056 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:34.273671 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59093: BlockingRecv error: recv got EOF from 127.0.98.2:59093 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 04:47:34 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:34 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:34.471401 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:34.465958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.3:58153)
0924 04:47:34.466262 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:34.466277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:34.466395 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:34.466644 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:34.466647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:34.466846 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:34.466937 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:34.466942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:34.467699 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:34.467702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:34.468323 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:34.468333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:34.468804 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:34.468807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:34.468875 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:34.469374 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:34.469386 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:34.471283 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":99,"threads_started":1}
W20260924 04:47:34.471534 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:34.466038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58153 (local address 127.0.98.62:35463)
0924 04:47:34.466245 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:34.466250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:34.466360 (+ 110us) server_negotiation.cc:407] Connection header received
0924 04:47:34.466483 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:34.466486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:34.466534 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:34.466608 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:34.467053 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:34.467586 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:34.468443 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:34.468677 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:34.468759 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:34.471420 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58153: BlockingRecv error: recv got EOF from 127.0.98.3:58153 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 24 04:47:35 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:35 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:35.109920 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:35.104457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.1:39055)
0924 04:47:35.104720 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:35.104734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:35.104818 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:35.105131 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:35.105134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:35.105356 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:35.105461 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:35.105471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:35.106275 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:35.106278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:35.106823 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:35.106831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:35.107348 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:35.107351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:35.107420 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:35.107957 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:35.107970 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:35.109798 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
W20260924 04:47:35.110191 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:35.104566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39055 (local address 127.0.98.62:35463)
0924 04:47:35.104824 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:35.104828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:35.104842 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:47:35.104881 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:35.104883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:35.104924 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:35.105009 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:35.105595 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:35.106144 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:35.106934 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:35.107228 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:35.107307 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:35.109957 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39055: BlockingRecv error: recv got EOF from 127.0.98.1:39055 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
Sep 24 04:47:35 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:47:35 dist-test-slave-hvcl krb5kdc[8545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:47:35.280392 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:35.274684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35463 (local address 127.0.98.2:41941)
0924 04:47:35.274963 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:35.275007 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:35.275134 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:35.275386 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:35.275388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:35.275554 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:35.275660 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:35.275665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:35.276462 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:35.276466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:35.277263 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:35.277275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:35.277759 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:35.277763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:35.277852 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:35.278391 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:35.278404 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:35.280233 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35463: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
W20260924 04:47:35.280473 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:35.274828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41941 (local address 127.0.98.62:35463)
0924 04:47:35.274988 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:35.274992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:35.275083 (+ 91us) server_negotiation.cc:407] Connection header received
0924 04:47:35.275241 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:35.275243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:35.275289 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:35.275354 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:35.275746 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:35.276309 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:35.277362 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:35.277642 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:35.277740 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:35.280374 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41941: BlockingRecv error: recv got EOF from 127.0.98.2:41941 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 04:47:35.370697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 9128
I20260924 04:47:35.375699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 9280
I20260924 04:47:35.380439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 9412
I20260924 04:47:35.384821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 9058
2026-09-24T04:47:35Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17926 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T04:47:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:35Z Disabled control of system clock
I20260924 04:47:35.412576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40213
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 04:47:35.521566 9608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:35.521853 9608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:35.521888 9608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:35.526198 9608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:35.526261 9608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:35.526275 9608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:35.526291 9608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:35.526309 9608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:47:35.532291 9608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:35.533605 9608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:35.535352 9608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:35.544126 9613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:35.544190 9616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:35.544194 9608 server_base.cc:1034] running on GCE node
W20260924 04:47:35.544139 9614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:35.544927 9608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:35.546020 9608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:35.547199 9608 hybrid_clock.cc:648] HybridClock initialized: now 1790225255547181 us; error 31 us; skew 500 ppm
I20260924 04:47:35.549093 9608 webserver.cc:460] Webserver started at http://127.0.98.62:34269/ using document root <none> and password file <none>
I20260924 04:47:35.549674 9608 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:35.549741 9608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:35.549926 9608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:35.552035 9608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "aadc1b26ab714a26969cfb747db96726"
format_stamp: "Formatted at 2026-09-24 04:47:35 on dist-test-slave-hvcl"
server_key: "2a0f0e841fa00e01c97333350d7e1fb3"
server_key_iv: "a16931a6f405a31b9ac199e09a867f17"
server_key_version: "encryptionkey@0"
I20260924 04:47:35.552578 9608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "aadc1b26ab714a26969cfb747db96726"
format_stamp: "Formatted at 2026-09-24 04:47:35 on dist-test-slave-hvcl"
server_key: "2a0f0e841fa00e01c97333350d7e1fb3"
server_key_iv: "a16931a6f405a31b9ac199e09a867f17"
server_key_version: "encryptionkey@0"
I20260924 04:47:35.556227 9608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:47:35.558593 9622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:35.559866 9608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:35.559985 9608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "aadc1b26ab714a26969cfb747db96726"
format_stamp: "Formatted at 2026-09-24 04:47:35 on dist-test-slave-hvcl"
server_key: "2a0f0e841fa00e01c97333350d7e1fb3"
server_key_iv: "a16931a6f405a31b9ac199e09a867f17"
server_key_version: "encryptionkey@0"
I20260924 04:47:35.560098 9608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:35.576670 9608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:35.577374 9608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:35.577551 9608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:35.583930 9608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40213
I20260924 04:47:35.583948 9674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40213 every 8 connection(s)
I20260924 04:47:35.584884 9608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:47:35.587960 9675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:35.588234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 9608
I20260924 04:47:35.588340 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:47:35.588644 392 external_mini_cluster.cc:1428] Setting key 002524ae358a242be359191f27543599
I20260924 04:47:35.594448 9675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726: Bootstrap starting.
I20260924 04:47:35.596728 9675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:35.597476 9675 log.cc:824] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:35.597805 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:35.590068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.0.98.62:40213)
0924 04:47:35.590751 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:35.590766 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:35.590802 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:47:35.591526 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:35.591545 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:35.591986 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:35.592300 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:35.592713 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:35.593647 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:35.594508 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:35.595059 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:35.595206 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:35.596068 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:35.596073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:35.596085 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:35.596476 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:35.596530 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:35.596550 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:35.596633 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:35.596775 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 04:47:35.597043 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":140,"threads_started":1}
I20260924 04:47:35.599525 9675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726: No bootstrap required, opened a new log
I20260924 04:47:35.602838 9675 raft_consensus.cc:348] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadc1b26ab714a26969cfb747db96726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40213 } }
I20260924 04:47:35.603096 9675 raft_consensus.cc:374] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:35.603143 9675 raft_consensus.cc:729] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aadc1b26ab714a26969cfb747db96726, State: Initialized, Role: FOLLOWER
I20260924 04:47:35.603550 9675 consensus_queue.cc:260] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadc1b26ab714a26969cfb747db96726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40213 } }
I20260924 04:47:35.603667 9675 raft_consensus.cc:388] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:47:35.603720 9675 raft_consensus.cc:482] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:47:35.603811 9675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:35.604722 9675 raft_consensus.cc:504] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadc1b26ab714a26969cfb747db96726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40213 } }
I20260924 04:47:35.605060 9675 leader_election.cc:302] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aadc1b26ab714a26969cfb747db96726; no voters:
I20260924 04:47:35.605376 9675 leader_election.cc:288] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:47:35.605471 9680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:35.605710 9680 raft_consensus.cc:686] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [term 1 LEADER]: Becoming Leader. State: Replica: aadc1b26ab714a26969cfb747db96726, State: Running, Role: LEADER
I20260924 04:47:35.606017 9680 consensus_queue.cc:237] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadc1b26ab714a26969cfb747db96726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40213 } }
I20260924 04:47:35.606524 9675 sys_catalog.cc:564] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:35.607635 9681 sys_catalog.cc:455] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aadc1b26ab714a26969cfb747db96726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadc1b26ab714a26969cfb747db96726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40213 } } }
I20260924 04:47:35.607658 9682 sys_catalog.cc:455] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aadc1b26ab714a26969cfb747db96726. Latest consensus state: current_term: 1 leader_uuid: "aadc1b26ab714a26969cfb747db96726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadc1b26ab714a26969cfb747db96726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40213 } } }
I20260924 04:47:35.607800 9681 sys_catalog.cc:458] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:35.607841 9682 sys_catalog.cc:458] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726 [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:35.608625 9689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:35.611272 9689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:35.616490 9689 catalog_manager.cc:1347] Generated new cluster ID: 9947086896444e55bbd58495931ae99d
I20260924 04:47:35.616554 9689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:35.629698 9689 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:47:35.630999 9689 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:35.635452 9689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aadc1b26ab714a26969cfb747db96726: Generated new TSK 0
I20260924 04:47:35.636190 9689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:47:35.647827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40213
--builtin_ntp_servers=127.0.98.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 04:47:35.756247 9699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:35.756505 9699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:35.756541 9699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:35.760437 9699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:35.760483 9699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:35.760533 9699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:47:35.765277 9699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:35.766289 9699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:35.767611 9699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:35.777305 9705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:35.777305 9704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:35.777357 9699 server_base.cc:1034] running on GCE node
W20260924 04:47:35.777305 9707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:35.777982 9699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:35.778614 9699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:35.779785 9699 hybrid_clock.cc:648] HybridClock initialized: now 1790225255779761 us; error 41 us; skew 500 ppm
I20260924 04:47:35.781616 9699 webserver.cc:460] Webserver started at http://127.0.98.1:36703/ using document root <none> and password file <none>
I20260924 04:47:35.782192 9699 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:35.782261 9699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:35.782446 9699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:35.784502 9699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "b341ec17df5d4a5e9bef74a25b413f08"
format_stamp: "Formatted at 2026-09-24 04:47:35 on dist-test-slave-hvcl"
server_key: "3030b7852192822c8bb7fba903b61620"
server_key_iv: "cb8d244a5b970e37dbf3ad2383a37ffe"
server_key_version: "encryptionkey@0"
I20260924 04:47:35.785048 9699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b341ec17df5d4a5e9bef74a25b413f08"
format_stamp: "Formatted at 2026-09-24 04:47:35 on dist-test-slave-hvcl"
server_key: "3030b7852192822c8bb7fba903b61620"
server_key_iv: "cb8d244a5b970e37dbf3ad2383a37ffe"
server_key_version: "encryptionkey@0"
I20260924 04:47:35.788749 9699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:47:35.791365 9713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:35.792673 9699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:35.792793 9699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "b341ec17df5d4a5e9bef74a25b413f08"
format_stamp: "Formatted at 2026-09-24 04:47:35 on dist-test-slave-hvcl"
server_key: "3030b7852192822c8bb7fba903b61620"
server_key_iv: "cb8d244a5b970e37dbf3ad2383a37ffe"
server_key_version: "encryptionkey@0"
I20260924 04:47:35.792902 9699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:35.812672 9699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:35.813390 9699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:35.813560 9699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:35.814136 9699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:35.815068 9699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:35.815122 9699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:35.815176 9699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:35.815201 9699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:35.823501 9699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36007
I20260924 04:47:35.823552 9826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36007 every 8 connection(s)
I20260924 04:47:35.824404 9699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:47:35.833761 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:35.826285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51167 (local address 127.0.98.62:40213)
0924 04:47:35.826442 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:35.826447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:35.827588 (+ 1141us) server_negotiation.cc:407] Connection header received
0924 04:47:35.828958 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:35.828961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:35.829010 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:35.829082 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:35.830137 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:35.830728 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:35.831898 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:35.832185 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:35.832283 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:35.833362 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:35.833364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:35.833366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:35.833379 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:35.833397 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:35.833405 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:35.833471 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:35.833577 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 04:47:35.833631 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:47:35.833859 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 9699
I20260924 04:47:35.833958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:47:35.834273 392 external_mini_cluster.cc:1428] Setting key 1a1a9daf0bb8a806a19dd183299c3c0a
I20260924 04:47:35.834879 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:35.826699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40213 (local address 127.0.98.1:51167)
0924 04:47:35.827444 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:35.827487 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:35.828722 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:35.829286 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:35.829296 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:35.829417 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:35.829971 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:35.829984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:35.830842 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:35.830845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:35.831775 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:35.831784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:35.832343 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:35.832347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:35.832501 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:35.833105 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:47:35.833106 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:35.833146 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:47:35.833212 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:35.833475 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:35.833480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:35.833483 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:47:35.833573 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:47:35.833868 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":109,"threads_started":1}
I20260924 04:47:35.836131 9827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40213
I20260924 04:47:35.836195 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40213
--builtin_ntp_servers=127.0.98.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 04:47:35.836369 9827 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:35.836872 9827 heartbeater.cc:511] Master 127.0.98.62:40213 requested a full tablet report, sending...
I20260924 04:47:35.838253 9639 ts_manager.cc:194] Registered new tserver with Master: b341ec17df5d4a5e9bef74a25b413f08 (127.0.98.1:36007)
I20260924 04:47:35.839787 9639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51167
W20260924 04:47:35.943581 9830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:35.943861 9830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:35.943898 9830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:35.947911 9830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:35.947970 9830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:35.948022 9830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:47:35.952741 9830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:35.953811 9830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:35.955176 9830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:47:35.965317 9830 server_base.cc:1034] running on GCE node
W20260924 04:47:35.965278 9835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:35.965257 9838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:35.965260 9836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:35.965937 9830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:35.966634 9830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:35.967825 9830 hybrid_clock.cc:648] HybridClock initialized: now 1790225255967797 us; error 42 us; skew 500 ppm
I20260924 04:47:35.969802 9830 webserver.cc:460] Webserver started at http://127.0.98.2:36063/ using document root <none> and password file <none>
I20260924 04:47:35.970387 9830 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:35.970454 9830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:35.970638 9830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:35.972725 9830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "60474fbe0182451a9d994249621300ae"
format_stamp: "Formatted at 2026-09-24 04:47:35 on dist-test-slave-hvcl"
server_key: "c260f682dcd19d89a72d34acdb397ab8"
server_key_iv: "1bf61edc39aad87db836a35bd3486063"
server_key_version: "encryptionkey@0"
I20260924 04:47:35.973279 9830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "60474fbe0182451a9d994249621300ae"
format_stamp: "Formatted at 2026-09-24 04:47:35 on dist-test-slave-hvcl"
server_key: "c260f682dcd19d89a72d34acdb397ab8"
server_key_iv: "1bf61edc39aad87db836a35bd3486063"
server_key_version: "encryptionkey@0"
I20260924 04:47:35.976974 9830 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 04:47:35.979573 9844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:35.980934 9830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:35.981047 9830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "60474fbe0182451a9d994249621300ae"
format_stamp: "Formatted at 2026-09-24 04:47:35 on dist-test-slave-hvcl"
server_key: "c260f682dcd19d89a72d34acdb397ab8"
server_key_iv: "1bf61edc39aad87db836a35bd3486063"
server_key_version: "encryptionkey@0"
I20260924 04:47:35.981154 9830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:36.007957 9830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:36.008677 9830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:36.008850 9830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:36.009439 9830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:36.010376 9830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:36.010423 9830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:36.010476 9830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:36.010504 9830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:36.018805 9830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38119
I20260924 04:47:36.018836 9957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38119 every 8 connection(s)
I20260924 04:47:36.019757 9830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:47:36.022738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 9830
I20260924 04:47:36.022850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:47:36.023126 392 external_mini_cluster.cc:1428] Setting key e84adca8f6fbb7a38d071e86f1135092
I20260924 04:47:36.025000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40213
--builtin_ntp_servers=127.0.98.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 04:47:36.029255 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:36.021557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60485 (local address 127.0.98.62:40213)
0924 04:47:36.021690 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:36.021694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:36.022771 (+ 1077us) server_negotiation.cc:407] Connection header received
0924 04:47:36.024397 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:36.024401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:36.024447 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:36.024515 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:36.025489 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:36.026008 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:36.027236 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:36.027471 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:36.027556 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:36.028783 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:36.028786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:36.028788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:36.028800 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:36.028821 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:36.028829 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:36.028895 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:36.029057 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 04:47:36.029124 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:47:36.030449 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:36.021944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40213 (local address 127.0.98.2:60485)
0924 04:47:36.022632 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:36.022682 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:36.024091 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:36.024694 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:36.024701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:36.024793 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:36.025361 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:36.025376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:36.026151 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:36.026155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:36.027119 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:36.027130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:36.027620 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:36.027624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:36.027813 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:36.028436 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:47:36.028438 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:36.028503 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:47:36.028594 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:36.028911 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:36.028920 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:36.028925 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:47:36.029060 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 04:47:36.029377 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":109,"threads_started":1}
I20260924 04:47:36.031709 9958 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40213
I20260924 04:47:36.031942 9958 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:36.032454 9958 heartbeater.cc:511] Master 127.0.98.62:40213 requested a full tablet report, sending...
I20260924 04:47:36.033623 9639 ts_manager.cc:194] Registered new tserver with Master: 60474fbe0182451a9d994249621300ae (127.0.98.2:38119)
I20260924 04:47:36.034221 9639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60485
W20260924 04:47:36.136297 9961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:36.136569 9961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:36.136620 9961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:36.140744 9961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:36.140810 9961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:36.140887 9961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:47:36.145731 9961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:36.146821 9961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:36.148178 9961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:47:36.157608 9961 server_base.cc:1034] running on GCE node
W20260924 04:47:36.157568 9966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:36.157555 9969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:36.157555 9967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:36.158237 9961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:36.158876 9961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:36.160038 9961 hybrid_clock.cc:648] HybridClock initialized: now 1790225256160017 us; error 32 us; skew 500 ppm
I20260924 04:47:36.161983 9961 webserver.cc:460] Webserver started at http://127.0.98.3:37827/ using document root <none> and password file <none>
I20260924 04:47:36.162550 9961 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:36.162616 9961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:36.162799 9961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:36.164973 9961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "0cbee8ae8f3f4273b61b60ee978996c7"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "e76f4b531ada915c14e63c43520aaf27"
server_key_iv: "d3358aed56f4dda3b1d27d9be8f509e7"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.165510 9961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0cbee8ae8f3f4273b61b60ee978996c7"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "e76f4b531ada915c14e63c43520aaf27"
server_key_iv: "d3358aed56f4dda3b1d27d9be8f509e7"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.169041 9961 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:47:36.171824 9975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:36.173080 9961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:36.173202 9961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "0cbee8ae8f3f4273b61b60ee978996c7"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "e76f4b531ada915c14e63c43520aaf27"
server_key_iv: "d3358aed56f4dda3b1d27d9be8f509e7"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.173307 9961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:36.186321 9961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:36.187052 9961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:36.187234 9961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:36.187798 9961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:36.188731 9961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:36.188778 9961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:36.188829 9961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:36.188858 9961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:36.196967 9961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33231
I20260924 04:47:36.197016 10088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33231 every 8 connection(s)
I20260924 04:47:36.197927 9961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:47:36.201164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 9961
I20260924 04:47:36.201283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:47:36.201548 392 external_mini_cluster.cc:1428] Setting key cd45617930f0bb763ecc16697820850d
I20260924 04:47:36.207602 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:36.199780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39891 (local address 127.0.98.62:40213)
0924 04:47:36.199905 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:36.199908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:36.200966 (+ 1058us) server_negotiation.cc:407] Connection header received
0924 04:47:36.202645 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:36.202647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:36.202700 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:36.202793 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:36.203858 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:36.204386 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:36.205563 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:36.205822 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:36.205912 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:36.207094 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:36.207098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:36.207100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:36.207113 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:36.207134 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:36.207142 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:36.207225 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:36.207430 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 04:47:36.207487 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:47:36.208514 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:36.200103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40213 (local address 127.0.98.3:39891)
0924 04:47:36.200798 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:36.200839 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:36.202430 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:36.203001 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:36.203011 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:36.203110 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:36.203730 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:36.203754 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:36.204511 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:36.204514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:36.205429 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:36.205438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:36.205980 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:36.205983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:36.206133 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:36.206780 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:47:36.206782 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:36.206838 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:47:36.206913 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:36.207278 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:36.207288 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:36.207295 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:47:36.207418 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 04:47:36.207660 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":120,"threads_started":1}
I20260924 04:47:36.209705 10089 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40213
I20260924 04:47:36.209971 10089 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:36.210507 10089 heartbeater.cc:511] Master 127.0.98.62:40213 requested a full tablet report, sending...
I20260924 04:47:36.211589 9639 ts_manager.cc:194] Registered new tserver with Master: 0cbee8ae8f3f4273b61b60ee978996c7 (127.0.98.3:33231)
I20260924 04:47:36.212167 9639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39891
I20260924 04:47:36.215479 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:47:36.218509 9678 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:36.217337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37030 (local address 127.0.98.62:40213)
0924 04:47:36.217578 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:36.217583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:36.217738 (+ 155us) server_negotiation.cc:407] Connection header received
0924 04:47:36.217819 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:36.217821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:36.217922 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:36.217999 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:36.218373 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37030: BlockingRecv error: recv got EOF from 127.0.0.1:37030 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 04:47:36.218555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 9699
I20260924 04:47:36.223593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 9830
I20260924 04:47:36.228024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 9961
I20260924 04:47:36.232050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 9608
2026-09-24T04:47:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (846 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T04:47:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:36Z Disabled control of system clock
I20260924 04:47:36.258649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42877
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:47:36.367312 10100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:36.367588 10100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:36.367626 10100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:36.371670 10100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:36.371729 10100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:36.371743 10100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:36.371758 10100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:36.371771 10100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:47:36.376591 10100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:36.377658 10100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:36.379073 10100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:36.387344 10106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:36.387290 10105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:36.387413 10100 server_base.cc:1034] running on GCE node
W20260924 04:47:36.387319 10108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:36.388123 10100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:36.389207 10100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:36.390362 10100 hybrid_clock.cc:648] HybridClock initialized: now 1790225256390344 us; error 32 us; skew 500 ppm
I20260924 04:47:36.397094 10100 webserver.cc:460] Webserver started at http://127.0.98.62:38587/ using document root <none> and password file <none>
I20260924 04:47:36.397698 10100 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:36.397763 10100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:36.397982 10100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:36.400148 10100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "1772338928dc48e7894c3627f7942bdd"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "7f80118455b6edc668e9876e2b7ede94"
server_key_iv: "bd36d485253f1d731be1aff36dc04cb4"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.400664 10100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "1772338928dc48e7894c3627f7942bdd"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "7f80118455b6edc668e9876e2b7ede94"
server_key_iv: "bd36d485253f1d731be1aff36dc04cb4"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.404320 10100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:47:36.406737 10114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:36.408129 10100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:36.408247 10100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "1772338928dc48e7894c3627f7942bdd"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "7f80118455b6edc668e9876e2b7ede94"
server_key_iv: "bd36d485253f1d731be1aff36dc04cb4"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.408365 10100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:36.412891 10100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:36.413489 10100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:36.413640 10100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:36.420460 10100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42877
I20260924 04:47:36.420462 10166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42877 every 8 connection(s)
I20260924 04:47:36.421451 10100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:47:36.424230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 10100
I20260924 04:47:36.424342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:47:36.424667 392 external_mini_cluster.cc:1428] Setting key 55aa3bae7f9cc7ec42c3ad440154f4be
I20260924 04:47:36.424926 10167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:36.430382 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:36.426163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.0.98.62:42877)
0924 04:47:36.426765 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:36.426780 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:36.426814 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:47:36.427548 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:36.427567 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:36.427931 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:36.428215 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:36.429034 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:36.429039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:36.429050 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:36.429060 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:36.429113 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:36.429129 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:36.429230 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:36.429341 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 04:47:36.429585 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
I20260924 04:47:36.432507 10167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd: Bootstrap starting.
I20260924 04:47:36.434684 10167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:36.435451 10167 log.cc:824] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:36.437633 10167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd: No bootstrap required, opened a new log
I20260924 04:47:36.441180 10167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1772338928dc48e7894c3627f7942bdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42877 } }
I20260924 04:47:36.441429 10167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:36.441488 10167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1772338928dc48e7894c3627f7942bdd, State: Initialized, Role: FOLLOWER
I20260924 04:47:36.441890 10167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1772338928dc48e7894c3627f7942bdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42877 } }
I20260924 04:47:36.442018 10167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:47:36.442080 10167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:47:36.442176 10167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:36.443120 10167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1772338928dc48e7894c3627f7942bdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42877 } }
I20260924 04:47:36.443488 10167 leader_election.cc:302] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1772338928dc48e7894c3627f7942bdd; no voters:
I20260924 04:47:36.443852 10167 leader_election.cc:288] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:47:36.443974 10172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:36.444221 10172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [term 1 LEADER]: Becoming Leader. State: Replica: 1772338928dc48e7894c3627f7942bdd, State: Running, Role: LEADER
I20260924 04:47:36.444542 10172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1772338928dc48e7894c3627f7942bdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42877 } }
I20260924 04:47:36.445060 10167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:36.446118 10173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1772338928dc48e7894c3627f7942bdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1772338928dc48e7894c3627f7942bdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42877 } } }
I20260924 04:47:36.446245 10173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:36.446458 10174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1772338928dc48e7894c3627f7942bdd. Latest consensus state: current_term: 1 leader_uuid: "1772338928dc48e7894c3627f7942bdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1772338928dc48e7894c3627f7942bdd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42877 } } }
I20260924 04:47:36.446559 10174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:36.446805 10181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:36.450290 10181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:36.455535 10181 catalog_manager.cc:1347] Generated new cluster ID: 38ac9668b31e4234ba649ed2889cbba7
I20260924 04:47:36.455605 10181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:36.468727 10181 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:47:36.469043 10181 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:36.477201 10181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1772338928dc48e7894c3627f7942bdd: Generated new TSK 0
I20260924 04:47:36.478025 10181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:47:36.489774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42877
--builtin_ntp_servers=127.0.98.20:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:47:36.613786 10191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:36.614070 10191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:36.614126 10191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:36.618270 10191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:36.618338 10191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:36.618425 10191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:47:36.623292 10191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:36.624369 10191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:36.625746 10191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:47:36.635210 10191 server_base.cc:1034] running on GCE node
W20260924 04:47:36.635178 10199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:36.635188 10197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:36.635308 10196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:36.635850 10191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:36.636484 10191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:36.637655 10191 hybrid_clock.cc:648] HybridClock initialized: now 1790225256637632 us; error 28 us; skew 500 ppm
I20260924 04:47:36.639439 10191 webserver.cc:460] Webserver started at http://127.0.98.1:45487/ using document root <none> and password file <none>
I20260924 04:47:36.639998 10191 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:36.640064 10191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:36.640249 10191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:36.642302 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "5afc6467adfd4641af8949912d9fbf93"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "d277432deb528e72f7d6621882faaf15"
server_key_iv: "259962ebb92ac7f11b96889a2cd314c5"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.642863 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5afc6467adfd4641af8949912d9fbf93"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "d277432deb528e72f7d6621882faaf15"
server_key_iv: "259962ebb92ac7f11b96889a2cd314c5"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.646695 10191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 04:47:36.649472 10205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:36.650883 10191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 04:47:36.651037 10191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "5afc6467adfd4641af8949912d9fbf93"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "d277432deb528e72f7d6621882faaf15"
server_key_iv: "259962ebb92ac7f11b96889a2cd314c5"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.651157 10191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:36.655790 10191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:36.656411 10191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:36.656579 10191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:36.657114 10191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:36.658054 10191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:36.658098 10191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:36.658135 10191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:36.658154 10191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:36.666117 10191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43447
I20260924 04:47:36.666172 10318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43447 every 8 connection(s)
I20260924 04:47:36.667047 10191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:47:36.672953 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:36.668817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60685 (local address 127.0.98.62:42877)
0924 04:47:36.668984 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:36.668989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:36.670112 (+ 1123us) server_negotiation.cc:407] Connection header received
0924 04:47:36.671465 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:36.671468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:36.671505 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:36.671584 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:36.672603 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:36.672606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:36.672608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:36.672615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:36.672628 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:36.672634 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:36.672695 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:36.672813 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 04:47:36.672846 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 04:47:36.673830 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:36.669250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42877 (local address 127.0.98.1:60685)
0924 04:47:36.669952 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:36.670006 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:36.671247 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:36.671749 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:36.671756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:36.671848 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:36.672370 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:47:36.672372 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:36.672420 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:47:36.672497 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:36.672688 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:36.672692 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:36.672715 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:47:36.672846 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 04:47:36.673047 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
I20260924 04:47:36.675107 10319 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42877
I20260924 04:47:36.675312 10319 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:36.675678 10319 heartbeater.cc:511] Master 127.0.98.62:42877 requested a full tablet report, sending...
I20260924 04:47:36.677121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 10191
I20260924 04:47:36.677238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:47:36.677186 10131 ts_manager.cc:194] Registered new tserver with Master: 5afc6467adfd4641af8949912d9fbf93 (127.0.98.1:43447)
I20260924 04:47:36.677573 392 external_mini_cluster.cc:1428] Setting key f85d6907c178a458ddfc4832a8d0853f
I20260924 04:47:36.679474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42877
--builtin_ntp_servers=127.0.98.20:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:47:36.788383 10322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:36.788692 10322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:36.788733 10322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:36.793033 10322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:36.793102 10322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:36.793160 10322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:47:36.798082 10322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:36.799208 10322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:36.800622 10322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:36.810256 10330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:36.810240 10327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:36.810237 10328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:36.810600 10322 server_base.cc:1034] running on GCE node
I20260924 04:47:36.811049 10322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:36.811756 10322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:36.812916 10322 hybrid_clock.cc:648] HybridClock initialized: now 1790225256812896 us; error 36 us; skew 500 ppm
I20260924 04:47:36.814675 10322 webserver.cc:460] Webserver started at http://127.0.98.2:45537/ using document root <none> and password file <none>
I20260924 04:47:36.815266 10322 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:36.815335 10322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:36.815527 10322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:36.817507 10322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "886fb35cfd6040719e41d177933cf3b9"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "1721a78941d3f08e5ed8de9e80facfcb"
server_key_iv: "3d767800c2cc3fa621073a57087a8cef"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.818071 10322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "886fb35cfd6040719e41d177933cf3b9"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "1721a78941d3f08e5ed8de9e80facfcb"
server_key_iv: "3d767800c2cc3fa621073a57087a8cef"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.821709 10322 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 04:47:36.824069 10336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:36.825311 10322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:47:36.825412 10322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "886fb35cfd6040719e41d177933cf3b9"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "1721a78941d3f08e5ed8de9e80facfcb"
server_key_iv: "3d767800c2cc3fa621073a57087a8cef"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.825515 10322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:36.829973 10322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:36.830546 10322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:36.830693 10322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:36.831234 10322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:36.832118 10322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:36.832163 10322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:36.832212 10322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:36.832238 10322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:36.840348 10322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41091
I20260924 04:47:36.840445 10449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41091 every 8 connection(s)
I20260924 04:47:36.841244 10322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:47:36.845062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 10322
I20260924 04:47:36.845172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:47:36.845431 392 external_mini_cluster.cc:1428] Setting key 3d0b8da36bf9daa474f2f4b4aad0e5e1
I20260924 04:47:36.847325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42877
--builtin_ntp_servers=127.0.98.20:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 04:47:36.847357 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:36.843063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58589 (local address 127.0.98.62:42877)
0924 04:47:36.843194 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:36.843198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:36.844255 (+ 1057us) server_negotiation.cc:407] Connection header received
0924 04:47:36.845691 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:36.845694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:36.845739 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:36.845812 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:36.846910 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:36.846913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:36.846916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:36.846929 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:36.846949 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:36.846956 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:36.847069 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:36.847186 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 04:47:36.847229 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:47:36.848206 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:36.843467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42877 (local address 127.0.98.2:58589)
0924 04:47:36.844111 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:36.844151 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:36.845445 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:36.845987 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:36.845995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:36.846092 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:36.846655 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:47:36.846657 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:36.846699 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:47:36.846770 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:36.847057 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:36.847065 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:36.847071 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:47:36.847227 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 04:47:36.847481 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":113,"threads_started":1}
I20260924 04:47:36.849318 10450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42877
I20260924 04:47:36.849557 10450 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:36.849969 10450 heartbeater.cc:511] Master 127.0.98.62:42877 requested a full tablet report, sending...
I20260924 04:47:36.851046 10131 ts_manager.cc:194] Registered new tserver with Master: 886fb35cfd6040719e41d177933cf3b9 (127.0.98.2:41091)
W20260924 04:47:36.957751 10453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:36.958029 10453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:36.958069 10453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:36.962186 10453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:36.962260 10453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:36.962317 10453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:47:36.967322 10453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:36.968451 10453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:36.969841 10453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:36.979696 10458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:36.979730 10459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:36.979730 10461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:36.980023 10453 server_base.cc:1034] running on GCE node
I20260924 04:47:36.980502 10453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:36.981133 10453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:36.982362 10453 hybrid_clock.cc:648] HybridClock initialized: now 1790225256982336 us; error 57 us; skew 500 ppm
I20260924 04:47:36.984329 10453 webserver.cc:460] Webserver started at http://127.0.98.3:41537/ using document root <none> and password file <none>
I20260924 04:47:36.984894 10453 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:36.984962 10453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:36.985153 10453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:36.987335 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "a379a703f4764affb365d7ae7c359ae1"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "5a2fa395e2a75901705bd8b76b054a2b"
server_key_iv: "0963de9f379aec8be00e67c4cc2305ca"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.987870 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a379a703f4764affb365d7ae7c359ae1"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "5a2fa395e2a75901705bd8b76b054a2b"
server_key_iv: "0963de9f379aec8be00e67c4cc2305ca"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.991673 10453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:47:36.994293 10467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:36.995739 10453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:47:36.995854 10453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "a379a703f4764affb365d7ae7c359ae1"
format_stamp: "Formatted at 2026-09-24 04:47:36 on dist-test-slave-hvcl"
server_key: "5a2fa395e2a75901705bd8b76b054a2b"
server_key_iv: "0963de9f379aec8be00e67c4cc2305ca"
server_key_version: "encryptionkey@0"
I20260924 04:47:36.995957 10453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:37.000659 10453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:37.001277 10453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:37.001432 10453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:37.001974 10453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:37.002889 10453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:37.002934 10453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:37.003005 10453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:37.003037 10453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:37.011369 10453 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38891
I20260924 04:47:37.011395 10580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38891 every 8 connection(s)
I20260924 04:47:37.012300 10453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:47:37.013278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 10453
I20260924 04:47:37.013392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:47:37.013734 392 external_mini_cluster.cc:1428] Setting key 700589bfc88d732b5a71f29d412f6001
I20260924 04:47:37.018554 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:37.014395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53803 (local address 127.0.98.62:42877)
0924 04:47:37.014548 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:37.014553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:37.015685 (+ 1132us) server_negotiation.cc:407] Connection header received
0924 04:47:37.017051 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:37.017054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:37.017091 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:37.017165 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:37.018172 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:37.018175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:37.018177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:47:37.018188 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:37.018210 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:47:37.018218 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:37.018306 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:37.018394 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 04:47:37.018433 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:47:37.019428 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:37.014872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42877 (local address 127.0.98.3:53803)
0924 04:47:37.015559 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:37.015600 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:37.016815 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:37.017301 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:37.017307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:37.017400 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:37.017899 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:47:37.017901 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:37.017965 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:47:37.018036 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:37.018294 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:37.018298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:37.018302 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:47:37.018421 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 04:47:37.018615 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":126,"threads_started":1}
I20260924 04:47:37.020500 10581 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42877
I20260924 04:47:37.020697 10581 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:37.021065 10581 heartbeater.cc:511] Master 127.0.98.62:42877 requested a full tablet report, sending...
I20260924 04:47:37.022001 10131 ts_manager.cc:194] Registered new tserver with Master: a379a703f4764affb365d7ae7c359ae1 (127.0.98.3:38891)
I20260924 04:47:37.027709 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:47:37.030431 10170 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:47:37.029595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.0.98.62:42877)
0924 04:47:37.029707 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:37.029711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:37.029850 (+ 139us) server_negotiation.cc:407] Connection header received
0924 04:47:37.029952 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:37.029954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:37.029984 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:37.030049 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:37.030303 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58276: BlockingRecv error: recv got EOF from 127.0.0.1:58276 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:47:37.030735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 10191
I20260924 04:47:37.035370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 10322
I20260924 04:47:37.039685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 10453
I20260924 04:47:37.043974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 10100
2026-09-24T04:47:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (811 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:47:37 dist-test-slave-hvcl krb5kdc[10591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:47:37 dist-test-slave-hvcl krb5kdc[10591](info): set up 2 sockets
Sep 24 04:47:37 dist-test-slave-hvcl krb5kdc[10591](info): commencing operation
krb5kdc: starting...
W20260924 04:47:38.011667 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:47:38 dist-test-slave-hvcl krb5kdc[10591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:47:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:47:38.130550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37413
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:38.237540 10604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:38.237838 10604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:38.237896 10604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:38.242012 10604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:47:38.242094 10604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:38.242120 10604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:38.242141 10604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:47:38.242170 10604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:47:38.247673 10604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:38.248783 10604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:38.249837 10604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:38.258802 10609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:38.258836 10604 server_base.cc:1034] running on GCE node
W20260924 04:47:38.258790 10610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:38.258802 10612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:38.259698 10604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:38.260885 10604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:38.262060 10604 hybrid_clock.cc:648] HybridClock initialized: now 1790225258262030 us; error 44 us; skew 500 ppm
Sep 24 04:47:38 dist-test-slave-hvcl krb5kdc[10591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:38.265077 10604 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:47:38.266098 10604 webserver.cc:460] Webserver started at http://127.0.98.62:41811/ using document root <none> and password file <none>
I20260924 04:47:38.266649 10604 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:38.266712 10604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:38.266907 10604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:38.269063 10604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "d1a32cc6ce114fdfa56501259965530a"
format_stamp: "Formatted at 2026-09-24 04:47:38 on dist-test-slave-hvcl"
server_key: "e0de3a2596b5d24da9ffdbad1a765106"
server_key_iv: "5970ba13c1c472d556a43af2cbbf1331"
server_key_version: "encryptionkey@0"
I20260924 04:47:38.269651 10604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "d1a32cc6ce114fdfa56501259965530a"
format_stamp: "Formatted at 2026-09-24 04:47:38 on dist-test-slave-hvcl"
server_key: "e0de3a2596b5d24da9ffdbad1a765106"
server_key_iv: "5970ba13c1c472d556a43af2cbbf1331"
server_key_version: "encryptionkey@0"
I20260924 04:47:38.273567 10604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 04:47:38.276058 10619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:38.277411 10604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:47:38.277540 10604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "d1a32cc6ce114fdfa56501259965530a"
format_stamp: "Formatted at 2026-09-24 04:47:38 on dist-test-slave-hvcl"
server_key: "e0de3a2596b5d24da9ffdbad1a765106"
server_key_iv: "5970ba13c1c472d556a43af2cbbf1331"
server_key_version: "encryptionkey@0"
I20260924 04:47:38.277644 10604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:38.290014 10604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:38.293027 10604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:38.293210 10604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:38.300185 10604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37413
I20260924 04:47:38.300441 10671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37413 every 8 connection(s)
I20260924 04:47:38.301142 10604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:47:38.303906 10672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:38.306567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 10604
I20260924 04:47:38.306697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:47:38.307041 392 external_mini_cluster.cc:1428] Setting key caf4100fbc9ff86783d5f187305c7b2c
I20260924 04:47:38.310129 10672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a: Bootstrap starting.
I20260924 04:47:38.312664 10672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:38.313555 10672 log.cc:824] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:38.315472 10672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a: No bootstrap required, opened a new log
Sep 24 04:47:38 dist-test-slave-hvcl krb5kdc[10591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:47:38.319383 10672 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a32cc6ce114fdfa56501259965530a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37413 } }
I20260924 04:47:38.319615 10672 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:38.319671 10672 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a32cc6ce114fdfa56501259965530a, State: Initialized, Role: FOLLOWER
I20260924 04:47:38.320101 10672 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a32cc6ce114fdfa56501259965530a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37413 } }
I20260924 04:47:38.320226 10672 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:47:38.320287 10672 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:47:38.320379 10672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:38.321219 10672 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a32cc6ce114fdfa56501259965530a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37413 } }
I20260924 04:47:38.321584 10672 leader_election.cc:302] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1a32cc6ce114fdfa56501259965530a; no voters:
I20260924 04:47:38.321885 10672 leader_election.cc:288] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:47:38.322010 10677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:38.322216 10677 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [term 1 LEADER]: Becoming Leader. State: Replica: d1a32cc6ce114fdfa56501259965530a, State: Running, Role: LEADER
I20260924 04:47:38.322472 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:38.308581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.0.98.62:37413)
0924 04:47:38.308991 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:38.309007 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:38.309043 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:47:38.309679 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:38.309695 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:38.310045 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:38.310389 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:38.311274 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:38.312365 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:38.313503 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:38.314029 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:38.314145 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:38.316682 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:38.316688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:38.316703 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:38.316734 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:38.319258 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:38.319746 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:38.319752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:38.319759 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:38.319888 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:38.320147 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:38.320151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:38.320153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:38.320670 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:38.320876 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:38.321171 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 04:47:38.321489 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
I20260924 04:47:38.322687 10677 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a32cc6ce114fdfa56501259965530a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37413 } }
I20260924 04:47:38.323098 10672 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:47:38.324252 10678 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a32cc6ce114fdfa56501259965530a. Latest consensus state: current_term: 1 leader_uuid: "d1a32cc6ce114fdfa56501259965530a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a32cc6ce114fdfa56501259965530a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37413 } } }
I20260924 04:47:38.324635 10678 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:38.324544 10679 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1a32cc6ce114fdfa56501259965530a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a32cc6ce114fdfa56501259965530a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37413 } } }
I20260924 04:47:38.324903 10679 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a [sys.catalog]: This master's current role is: LEADER
I20260924 04:47:38.325150 10690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:47:38.328075 10690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:47:38.333735 10690 catalog_manager.cc:1347] Generated new cluster ID: b64f404d7b18487181e80a78f00f4e04
I20260924 04:47:38.333814 10690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:47:38.338912 10690 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:47:38.339846 10690 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:47:38.345297 10690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a: Generated new TSK 0
I20260924 04:47:38.345952 10690 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:47:38.378958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37413
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:38.491986 10698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:38.492268 10698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:38.492329 10698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:38.496405 10698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:38.496474 10698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:38.496553 10698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:47:38.501865 10698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:38.502915 10698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:38.503957 10698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:38.513500 10706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:38.513549 10704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:38.513582 10698 server_base.cc:1034] running on GCE node
W20260924 04:47:38.513620 10703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:38.514171 10698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:38.514811 10698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:38.515966 10698 hybrid_clock.cc:648] HybridClock initialized: now 1790225258515937 us; error 45 us; skew 500 ppm
Sep 24 04:47:38 dist-test-slave-hvcl krb5kdc[10591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:38.518692 10698 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:47:38.519569 10698 webserver.cc:460] Webserver started at http://127.0.98.1:36079/ using document root <none> and password file <none>
I20260924 04:47:38.520090 10698 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:38.520150 10698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:38.520347 10698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:38.522305 10698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f1fe0379e5a40b4924b5d96d26e32d1"
format_stamp: "Formatted at 2026-09-24 04:47:38 on dist-test-slave-hvcl"
server_key: "919e045f88b00233ce023bdd68c58fdc"
server_key_iv: "1f26b1aeba457e529c26db39a2868c70"
server_key_version: "encryptionkey@0"
I20260924 04:47:38.522830 10698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f1fe0379e5a40b4924b5d96d26e32d1"
format_stamp: "Formatted at 2026-09-24 04:47:38 on dist-test-slave-hvcl"
server_key: "919e045f88b00233ce023bdd68c58fdc"
server_key_iv: "1f26b1aeba457e529c26db39a2868c70"
server_key_version: "encryptionkey@0"
I20260924 04:47:38.526556 10698 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:47:38.528971 10713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:38.530232 10698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:47:38.530357 10698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "8f1fe0379e5a40b4924b5d96d26e32d1"
format_stamp: "Formatted at 2026-09-24 04:47:38 on dist-test-slave-hvcl"
server_key: "919e045f88b00233ce023bdd68c58fdc"
server_key_iv: "1f26b1aeba457e529c26db39a2868c70"
server_key_version: "encryptionkey@0"
I20260924 04:47:38.530452 10698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:38.558178 10698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:38.561403 10698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:38.561589 10698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:38.562155 10698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:38.562997 10698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:38.563042 10698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:38.563097 10698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:38.563126 10698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:38.571261 10698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37085
I20260924 04:47:38.571292 10826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37085 every 8 connection(s)
I20260924 04:47:38.572132 10698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:47:38.575582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 10698
I20260924 04:47:38.575672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:47:38.575910 392 external_mini_cluster.cc:1428] Setting key bbb42e75a29a2819e42811f742efa5f6
Sep 24 04:47:38 dist-test-slave-hvcl krb5kdc[10591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:47:38.585810 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:38.573840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51879 (local address 127.0.98.62:37413)
0924 04:47:38.573974 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:38.573978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:38.575034 (+ 1056us) server_negotiation.cc:407] Connection header received
0924 04:47:38.575968 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:38.575971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:38.576029 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:38.576099 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:38.577239 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:38.577755 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:38.579154 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:38.579432 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:38.579519 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:38.582408 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:38.582412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:38.582437 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:38.582464 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:38.583883 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:38.584471 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:38.584475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:38.584477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:38.584560 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:38.584859 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:38.584861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:38.584862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:38.585033 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:38.585099 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:38.585583 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 04:47:38.585686 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:47:38.586663 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:38.574241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37413 (local address 127.0.98.1:51879)
0924 04:47:38.574861 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:38.574901 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:38.575770 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:38.576276 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:38.576283 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:38.576629 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:38.577105 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:38.577118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:38.577866 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:38.577868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:38.579040 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:38.579051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:38.579577 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:38.579580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:38.579765 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:38.580345 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:38.580365 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:38.582199 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:38.584013 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:38.584020 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:38.584037 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:38.584365 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:38.584643 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:38.584646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:38.584647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:38.584790 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:38.585106 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:38.585111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:38.585381 (+ 270us) client_negotiation.cc:769] Sending connection context
0924 04:47:38.585541 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 04:47:38.585802 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:47:38.587929 10827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37413
I20260924 04:47:38.588157 10827 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:38.588644 10827 heartbeater.cc:511] Master 127.0.98.62:37413 requested a full tablet report, sending...
I20260924 04:47:38.590088 10636 ts_manager.cc:194] Registered new tserver with Master: 8f1fe0379e5a40b4924b5d96d26e32d1 (127.0.98.1:37085)
I20260924 04:47:38.591303 10636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:47:38.601297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37413
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:38.708838 10832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:38.709105 10832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:38.709147 10832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:38.713143 10832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:38.713212 10832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:38.713269 10832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:47:38.718592 10832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:38.719619 10832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:38.720585 10832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:47:38.729889 10837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:38.730007 10832 server_base.cc:1034] running on GCE node
W20260924 04:47:38.730017 10838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:38.729933 10840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:38.730588 10832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:38.731254 10832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:38.732462 10832 hybrid_clock.cc:648] HybridClock initialized: now 1790225258732438 us; error 40 us; skew 500 ppm
Sep 24 04:47:38 dist-test-slave-hvcl krb5kdc[10591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:38.735214 10832 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:47:38.736068 10832 webserver.cc:460] Webserver started at http://127.0.98.2:37813/ using document root <none> and password file <none>
I20260924 04:47:38.736579 10832 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:38.736632 10832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:38.736791 10832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:38.738713 10832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "eded914d5391420391a5e67e9af178e0"
format_stamp: "Formatted at 2026-09-24 04:47:38 on dist-test-slave-hvcl"
server_key: "02fe9d6b279dc389dc1c7cbc4c2eb86e"
server_key_iv: "534584bdce23e9df465fb6a6c59fcb79"
server_key_version: "encryptionkey@0"
I20260924 04:47:38.739224 10832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eded914d5391420391a5e67e9af178e0"
format_stamp: "Formatted at 2026-09-24 04:47:38 on dist-test-slave-hvcl"
server_key: "02fe9d6b279dc389dc1c7cbc4c2eb86e"
server_key_iv: "534584bdce23e9df465fb6a6c59fcb79"
server_key_version: "encryptionkey@0"
I20260924 04:47:38.742666 10832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:47:38.745029 10847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:38.746263 10832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:47:38.746395 10832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "eded914d5391420391a5e67e9af178e0"
format_stamp: "Formatted at 2026-09-24 04:47:38 on dist-test-slave-hvcl"
server_key: "02fe9d6b279dc389dc1c7cbc4c2eb86e"
server_key_iv: "534584bdce23e9df465fb6a6c59fcb79"
server_key_version: "encryptionkey@0"
I20260924 04:47:38.746490 10832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:38.759624 10832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:38.762707 10832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:38.762871 10832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:38.763495 10832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:38.764272 10832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:38.764317 10832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:38.764366 10832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:38.764395 10832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:38.772722 10832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45077
I20260924 04:47:38.772751 10960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45077 every 8 connection(s)
I20260924 04:47:38.773599 10832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:47:38.777127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 10832
I20260924 04:47:38.777215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:47:38.777477 392 external_mini_cluster.cc:1428] Setting key 28d4b7410db7e9a3f636569666049244
Sep 24 04:47:38 dist-test-slave-hvcl krb5kdc[10591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:47:38.786841 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:38.775326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39565 (local address 127.0.98.62:37413)
0924 04:47:38.775488 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:38.775492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:38.776485 (+ 993us) server_negotiation.cc:407] Connection header received
0924 04:47:38.777381 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:38.777385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:38.777440 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:38.777511 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:38.778999 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:38.779517 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:38.780699 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:38.780916 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:38.781004 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:38.783470 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:38.783473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:38.783475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:38.783500 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:38.784802 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:38.785305 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:38.785309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:38.785310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:38.785387 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:38.785710 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:38.785713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:38.785715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:38.785925 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:38.786013 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:38.786536 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 04:47:38.786671 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:47:38.787828 10963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:38.775703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37413 (local address 127.0.98.2:39565)
0924 04:47:38.776328 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:38.776368 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:38.777168 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:38.777707 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:38.777715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:38.778157 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:38.778826 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:38.778847 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:38.779640 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:38.779643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:38.780592 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:38.780602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:38.781052 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:38.781056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:38.781232 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:38.781813 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:38.781829 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:38.783308 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:38.784913 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:38.784918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:38.784935 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:38.785211 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:38.785468 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:38.785471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:38.785473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:38.785624 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:38.786023 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:38.786030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:38.786296 (+ 266us) client_negotiation.cc:769] Sending connection context
0924 04:47:38.786478 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 04:47:38.786873 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:47:38.789072 10961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37413
I20260924 04:47:38.789315 10961 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:38.789808 10961 heartbeater.cc:511] Master 127.0.98.62:37413 requested a full tablet report, sending...
I20260924 04:47:38.790813 10636 ts_manager.cc:194] Registered new tserver with Master: eded914d5391420391a5e67e9af178e0 (127.0.98.2:45077)
I20260924 04:47:38.791395 10636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:47:38.803717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37413
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:47:38.911903 10966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:47:38.912163 10966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:47:38.912204 10966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:47:38.916170 10966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:47:38.916239 10966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:47:38.916298 10966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:47:38.921727 10966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:47:38.922822 10966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:47:38.923947 10966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:47:38.934015 10966 server_base.cc:1034] running on GCE node
W20260924 04:47:38.933976 10971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:38.933976 10972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:47:38.934007 10974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:47:38.934639 10966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:47:38.935434 10966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:47:38.936601 10966 hybrid_clock.cc:648] HybridClock initialized: now 1790225258936582 us; error 40 us; skew 500 ppm
Sep 24 04:47:38 dist-test-slave-hvcl krb5kdc[10591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:47:38.939414 10966 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:47:38.940348 10966 webserver.cc:460] Webserver started at http://127.0.98.3:44391/ using document root <none> and password file <none>
I20260924 04:47:38.940903 10966 fs_manager.cc:359] Metadata directory not provided
I20260924 04:47:38.940968 10966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:47:38.941165 10966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:47:38.943217 10966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "d863b7eb46ef4490b602c5f6af5001a7"
format_stamp: "Formatted at 2026-09-24 04:47:38 on dist-test-slave-hvcl"
server_key: "7395d243921f55f71d52811cbb3e71d8"
server_key_iv: "c6ead430e78884e40e176f073cb29f7e"
server_key_version: "encryptionkey@0"
I20260924 04:47:38.943749 10966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d863b7eb46ef4490b602c5f6af5001a7"
format_stamp: "Formatted at 2026-09-24 04:47:38 on dist-test-slave-hvcl"
server_key: "7395d243921f55f71d52811cbb3e71d8"
server_key_iv: "c6ead430e78884e40e176f073cb29f7e"
server_key_version: "encryptionkey@0"
I20260924 04:47:38.947456 10966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:47:38.949913 10981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:38.951314 10966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:47:38.951423 10966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "d863b7eb46ef4490b602c5f6af5001a7"
format_stamp: "Formatted at 2026-09-24 04:47:38 on dist-test-slave-hvcl"
server_key: "7395d243921f55f71d52811cbb3e71d8"
server_key_iv: "c6ead430e78884e40e176f073cb29f7e"
server_key_version: "encryptionkey@0"
I20260924 04:47:38.951538 10966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:47:38.987653 10966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:47:38.990780 10966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:47:38.990989 10966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:47:38.991575 10966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:47:38.992394 10966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:47:38.992440 10966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:38.992492 10966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:47:38.992520 10966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:47:39.000872 10966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39921
I20260924 04:47:39.000900 11094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39921 every 8 connection(s)
I20260924 04:47:39.001787 10966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:47:39.010565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 10966
I20260924 04:47:39.010694 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:47:39.011055 392 external_mini_cluster.cc:1428] Setting key 59bff869b8357fdd3778ab3691145bf2
Sep 24 04:47:39 dist-test-slave-hvcl krb5kdc[10591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:47:39.015365 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.003721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51807 (local address 127.0.98.62:37413)
0924 04:47:39.003916 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:39.003921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:39.005086 (+ 1165us) server_negotiation.cc:407] Connection header received
0924 04:47:39.005932 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:39.005936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:39.005983 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:39.006067 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:39.007370 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.007903 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.009089 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.009315 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.009401 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:39.012036 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:39.012039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:39.012041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:39.012065 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:39.013490 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:39.013946 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:39.013949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:39.013951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:39.014025 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:39.014325 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:39.014328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:39.014330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:39.014506 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:39.014590 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:39.015086 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 04:47:39.015206 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 04:47:39.016216 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.004141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37413 (local address 127.0.98.3:51807)
0924 04:47:39.004895 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:39.004937 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:39.005729 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:39.006225 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:39.006232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:39.006664 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:47:39.007222 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:39.007238 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.008024 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.008028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:39.008978 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:39.008988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.009491 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.009494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:39.009638 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:39.010191 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:47:39.010212 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:47:39.011888 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:47:39.013594 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:39.013600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:39.013616 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:39.013856 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:39.014102 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:47:39.014104 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:47:39.014107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:47:39.014241 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:47:39.014600 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:47:39.014607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:47:39.014852 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 04:47:39.015048 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 04:47:39.015396 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":138,"threads_started":1}
I20260924 04:47:39.017272 11095 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37413
I20260924 04:47:39.017488 11095 heartbeater.cc:464] Registering TS with master...
I20260924 04:47:39.017923 11095 heartbeater.cc:511] Master 127.0.98.62:37413 requested a full tablet report, sending...
I20260924 04:47:39.018837 10636 ts_manager.cc:194] Registered new tserver with Master: d863b7eb46ef4490b602c5f6af5001a7 (127.0.98.3:39921)
I20260924 04:47:39.019481 10636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51807
I20260924 04:47:39.024799 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:47:39.033569 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.026651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.0.98.62:37413)
0924 04:47:39.026786 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:39.026791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:39.026926 (+ 135us) server_negotiation.cc:407] Connection header received
0924 04:47:39.027076 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:39.027079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:39.027119 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:39.027176 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:47:39.027824 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.028329 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.029152 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.029376 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.029458 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:39.030704 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:47:39.030708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:47:39.030710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:47:39.030733 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:47:39.032089 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:39.032459 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:39.032463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:39.032465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:39.032537 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:47:39.032769 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:47:39.032772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:47:39.032774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:47:39.032989 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:47:39.033109 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:39.033317 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 04:47:39.033419 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:47:39.036507 10636 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:47:39.039533 10636 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:47:39.053977 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.048472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.0.98.1:37085)
0924 04:47:39.048787 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:39.048803 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:39.049315 (+ 512us) server_negotiation.cc:407] Connection header received
0924 04:47:39.049394 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:39.049399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:39.049639 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:39.049785 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:39.049932 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.051119 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.052570 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.053225 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.053322 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:39.053472 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:39.053714 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 04:47:39.053819 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":55,"threads_started":1}
I20260924 04:47:39.054021 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.048160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33342 (local address 127.0.98.2:45077)
0924 04:47:39.048972 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:39.048984 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:39.049006 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:47:39.049095 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:39.049100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:39.049366 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:39.049518 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:39.049805 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.050837 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.052607 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.053315 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.053420 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:39.053553 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:39.053809 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 04:47:39.053899 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":150,"threads_started":1}
I20260924 04:47:39.054378 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.048770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39921 (local address 127.0.0.1:48920)
0924 04:47:39.049544 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:39.049556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:39.049627 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:39.050241 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:39.050244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:39.050261 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:39.050441 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:39.050450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.051676 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.051679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:39.052757 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:39.052768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.053747 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.053751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:39.053836 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:39.053858 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:47:39.053933 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 04:47:39.054022 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":70,"thread_start_us":81,"threads_started":1}
I20260924 04:47:39.054409 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.049151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.0.98.3:39921)
0924 04:47:39.049809 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:39.049819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:39.049841 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:47:39.049910 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:39.049915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:39.050104 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:39.050254 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:39.050553 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.051532 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.052879 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.053631 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.053723 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:39.053842 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:39.053969 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 04:47:39.054061 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":135,"threads_started":1}
I20260924 04:47:39.054382 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.048007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45077 (local address 127.0.0.1:33342)
0924 04:47:39.048715 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:39.048766 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:39.048902 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:39.049515 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:39.049521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:39.049543 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:39.049684 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:39.049691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.050960 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.050962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:39.052521 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:39.052529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.053559 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.053562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:39.053642 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:39.053688 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:47:39.054158 (+ 470us) client_negotiation.cc:240] Negotiation successful
0924 04:47:39.054247 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":183,"thread_start_us":86,"threads_started":1}
I20260924 04:47:39.054611 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.048287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37085 (local address 127.0.0.1:60596)
0924 04:47:39.049132 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:39.049144 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:39.049225 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:39.049763 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:39.049766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:39.049780 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:39.049851 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:39.049857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.051243 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.051246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:39.052443 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:39.052453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.053337 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.053340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:39.053457 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:39.053495 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 04:47:39.053599 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:47:39.054215 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":506,"thread_start_us":94,"threads_started":1}
I20260924 04:47:39.057418 11029 tablet_service.cc:1467] Processing CreateTablet for tablet 200a5640a4694a06b52e459663473df6 (DEFAULT_TABLE table=test-table [id=cf31c30733a846c3844523b5acb73db0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:39.057705 10895 tablet_service.cc:1467] Processing CreateTablet for tablet 200a5640a4694a06b52e459663473df6 (DEFAULT_TABLE table=test-table [id=cf31c30733a846c3844523b5acb73db0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:39.057705 10761 tablet_service.cc:1467] Processing CreateTablet for tablet 200a5640a4694a06b52e459663473df6 (DEFAULT_TABLE table=test-table [id=cf31c30733a846c3844523b5acb73db0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:47:39.058434 11029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200a5640a4694a06b52e459663473df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:39.058744 10761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200a5640a4694a06b52e459663473df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:39.058743 10895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 200a5640a4694a06b52e459663473df6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:47:39.064841 11112 tablet_bootstrap.cc:492] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7: Bootstrap starting.
I20260924 04:47:39.066255 11113 tablet_bootstrap.cc:492] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0: Bootstrap starting.
I20260924 04:47:39.066255 11114 tablet_bootstrap.cc:492] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1: Bootstrap starting.
I20260924 04:47:39.067286 11112 tablet_bootstrap.cc:654] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:39.068363 11112 log.cc:824] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:39.068503 11114 tablet_bootstrap.cc:654] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:39.068945 11113 tablet_bootstrap.cc:654] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0: Neither blocks nor log segments found. Creating new log.
I20260924 04:47:39.069826 11113 log.cc:824] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:39.069826 11114 log.cc:824] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:47:39.070408 11112 tablet_bootstrap.cc:492] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7: No bootstrap required, opened a new log
I20260924 04:47:39.070612 11112 ts_tablet_manager.cc:1397] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:47:39.071625 11114 tablet_bootstrap.cc:492] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1: No bootstrap required, opened a new log
I20260924 04:47:39.071805 11114 ts_tablet_manager.cc:1397] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 04:47:39.071830 11113 tablet_bootstrap.cc:492] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0: No bootstrap required, opened a new log
I20260924 04:47:39.072016 11113 ts_tablet_manager.cc:1397] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 04:47:39.074484 11112 raft_consensus.cc:348] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eded914d5391420391a5e67e9af178e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "d863b7eb46ef4490b602c5f6af5001a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39921 } } peers { permanent_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37085 } }
I20260924 04:47:39.074604 11114 raft_consensus.cc:348] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eded914d5391420391a5e67e9af178e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "d863b7eb46ef4490b602c5f6af5001a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39921 } } peers { permanent_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37085 } }
I20260924 04:47:39.074770 11112 raft_consensus.cc:374] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:39.074801 11114 raft_consensus.cc:374] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:39.074826 11112 raft_consensus.cc:729] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d863b7eb46ef4490b602c5f6af5001a7, State: Initialized, Role: FOLLOWER
I20260924 04:47:39.074851 11114 raft_consensus.cc:729] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1fe0379e5a40b4924b5d96d26e32d1, State: Initialized, Role: FOLLOWER
I20260924 04:47:39.075287 11114 consensus_queue.cc:260] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eded914d5391420391a5e67e9af178e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "d863b7eb46ef4490b602c5f6af5001a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39921 } } peers { permanent_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37085 } }
I20260924 04:47:39.075294 11112 consensus_queue.cc:260] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eded914d5391420391a5e67e9af178e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "d863b7eb46ef4490b602c5f6af5001a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39921 } } peers { permanent_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37085 } }
I20260924 04:47:39.075592 11113 raft_consensus.cc:348] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eded914d5391420391a5e67e9af178e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "d863b7eb46ef4490b602c5f6af5001a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39921 } } peers { permanent_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37085 } }
I20260924 04:47:39.075824 11113 raft_consensus.cc:374] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:47:39.075894 11113 raft_consensus.cc:729] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eded914d5391420391a5e67e9af178e0, State: Initialized, Role: FOLLOWER
I20260924 04:47:39.076261 11095 heartbeater.cc:503] Master 127.0.98.62:37413 was elected leader, sending a full tablet report...
I20260924 04:47:39.076282 11112 ts_tablet_manager.cc:1428] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:47:39.076257 11113 consensus_queue.cc:260] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eded914d5391420391a5e67e9af178e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "d863b7eb46ef4490b602c5f6af5001a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39921 } } peers { permanent_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37085 } }
I20260924 04:47:39.076330 11114 ts_tablet_manager.cc:1428] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 04:47:39.076829 10961 heartbeater.cc:503] Master 127.0.98.62:37413 was elected leader, sending a full tablet report...
I20260924 04:47:39.077070 11113 ts_tablet_manager.cc:1428] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:47:39.077525 10827 heartbeater.cc:503] Master 127.0.98.62:37413 was elected leader, sending a full tablet report...
W20260924 04:47:39.252728 11096 tablet.cc:2369] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:47:39.274699 10962 tablet.cc:2369] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:47:39.305053 11118 raft_consensus.cc:482] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:47:39.305301 11118 raft_consensus.cc:504] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eded914d5391420391a5e67e9af178e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "d863b7eb46ef4490b602c5f6af5001a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39921 } } peers { permanent_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37085 } }
I20260924 04:47:39.306501 11118 leader_election.cc:288] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eded914d5391420391a5e67e9af178e0 (127.0.98.2:45077), d863b7eb46ef4490b602c5f6af5001a7 (127.0.98.3:39921)
I20260924 04:47:39.310776 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.307331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43077 (local address 127.0.98.3:39921)
0924 04:47:39.307460 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:39.307465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:39.307601 (+ 136us) server_negotiation.cc:407] Connection header received
0924 04:47:39.307709 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:39.307712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:39.307755 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:39.307817 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:39.308068 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.308551 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.309648 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.310058 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.310158 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:39.310225 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:39.310567 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 04:47:39.310647 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:47:39.311138 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.307210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39921 (local address 127.0.98.1:43077)
0924 04:47:39.307517 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:39.307529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:39.307625 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:39.307848 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:39.307851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:39.307865 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:39.307969 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:39.307973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.308661 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.308664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:39.309515 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:39.309524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.310283 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.310287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:39.310409 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:39.310433 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:47:39.310925 (+ 492us) client_negotiation.cc:240] Negotiation successful
0924 04:47:39.311025 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260924 04:47:39.311481 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.306875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37423 (local address 127.0.98.2:45077)
0924 04:47:39.307043 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:39.307048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:39.307208 (+ 160us) server_negotiation.cc:407] Connection header received
0924 04:47:39.307430 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:39.307434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:39.307482 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:39.307527 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:47:39.307876 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.308654 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.310137 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.310704 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.310798 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:39.310870 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:39.311280 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 04:47:39.311355 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:47:39.311774 11049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200a5640a4694a06b52e459663473df6" candidate_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d863b7eb46ef4490b602c5f6af5001a7" is_pre_election: true
I20260924 04:47:39.311858 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.306758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45077 (local address 127.0.98.1:37423)
0924 04:47:39.307109 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:39.307125 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:47:39.307294 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:47:39.307648 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:47:39.307651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:47:39.307666 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:47:39.307773 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:39.307779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.308782 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.308785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:39.309982 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:47:39.309993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.310809 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.310812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:47:39.311140 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:39.311164 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:47:39.311654 (+ 490us) client_negotiation.cc:240] Negotiation successful
0924 04:47:39.311727 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
I20260924 04:47:39.312088 11049 raft_consensus.cc:2454] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f1fe0379e5a40b4924b5d96d26e32d1 in term 0.
I20260924 04:47:39.312393 10915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200a5640a4694a06b52e459663473df6" candidate_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eded914d5391420391a5e67e9af178e0" is_pre_election: true
I20260924 04:47:39.312557 10716 leader_election.cc:302] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1fe0379e5a40b4924b5d96d26e32d1, d863b7eb46ef4490b602c5f6af5001a7; no voters:
I20260924 04:47:39.312671 10915 raft_consensus.cc:2454] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f1fe0379e5a40b4924b5d96d26e32d1 in term 0.
I20260924 04:47:39.312850 11118 raft_consensus.cc:2781] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:47:39.312951 11118 raft_consensus.cc:482] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:47:39.313006 11118 raft_consensus.cc:3037] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:39.314036 11118 raft_consensus.cc:504] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eded914d5391420391a5e67e9af178e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "d863b7eb46ef4490b602c5f6af5001a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39921 } } peers { permanent_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37085 } }
I20260924 04:47:39.314349 11118 leader_election.cc:288] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [CANDIDATE]: Term 1 election: Requested vote from peers eded914d5391420391a5e67e9af178e0 (127.0.98.2:45077), d863b7eb46ef4490b602c5f6af5001a7 (127.0.98.3:39921)
I20260924 04:47:39.314714 10915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200a5640a4694a06b52e459663473df6" candidate_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eded914d5391420391a5e67e9af178e0"
I20260924 04:47:39.314752 11049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "200a5640a4694a06b52e459663473df6" candidate_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d863b7eb46ef4490b602c5f6af5001a7"
I20260924 04:47:39.314836 10915 raft_consensus.cc:3037] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:39.314855 11049 raft_consensus.cc:3037] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:47:39.315878 11049 raft_consensus.cc:2454] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f1fe0379e5a40b4924b5d96d26e32d1 in term 1.
I20260924 04:47:39.315883 10915 raft_consensus.cc:2454] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f1fe0379e5a40b4924b5d96d26e32d1 in term 1.
I20260924 04:47:39.316134 10716 leader_election.cc:302] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1fe0379e5a40b4924b5d96d26e32d1, d863b7eb46ef4490b602c5f6af5001a7; no voters:
I20260924 04:47:39.316310 11118 raft_consensus.cc:2781] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:47:39.316507 11118 raft_consensus.cc:686] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [term 1 LEADER]: Becoming Leader. State: Replica: 8f1fe0379e5a40b4924b5d96d26e32d1, State: Running, Role: LEADER
I20260924 04:47:39.316854 11118 consensus_queue.cc:237] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eded914d5391420391a5e67e9af178e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } } peers { permanent_uuid: "d863b7eb46ef4490b602c5f6af5001a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39921 } } peers { permanent_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37085 } }
I20260924 04:47:39.319829 10634 catalog_manager.cc:5495] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f1fe0379e5a40b4924b5d96d26e32d1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eded914d5391420391a5e67e9af178e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d863b7eb46ef4490b602c5f6af5001a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f1fe0379e5a40b4924b5d96d26e32d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37085 } health_report { overall_health: HEALTHY } } }
W20260924 04:47:39.323367 10828 tablet.cc:2369] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:47:39.346045 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.342509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.0.98.1:37085)
0924 04:47:39.342632 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:47:39.342636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:47:39.342697 (+ 61us) server_negotiation.cc:407] Connection header received
0924 04:47:39.342856 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:47:39.342859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:47:39.342923 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:47:39.343051 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:47:39.343297 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.343783 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.344629 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:47:39.344857 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:47:39.344949 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:47:39.345251 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:47:39.345672 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:47:39.345779 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:47:39.345840 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 04:47:39.345906 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:47:39.351931 10915 raft_consensus.cc:1261] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0 [term 1 FOLLOWER]: Refusing update from remote peer 8f1fe0379e5a40b4924b5d96d26e32d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:39.351962 11049 raft_consensus.cc:1261] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7 [term 1 FOLLOWER]: Refusing update from remote peer 8f1fe0379e5a40b4924b5d96d26e32d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:47:39.352583 11118 consensus_queue.cc:1035] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eded914d5391420391a5e67e9af178e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:39.352784 11123 consensus_queue.cc:1035] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d863b7eb46ef4490b602c5f6af5001a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:47:39.362040 11127 mvcc.cc:205] Tried to move back new op lower bound from 7332762662298411008 to 7332762662164287488. Current Snapshot: MvccSnapshot[applied={T|T < 7332762662298411008}]
I20260924 04:47:39.369915 10634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:39.370085 10634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:47:39.372356 10634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1a32cc6ce114fdfa56501259965530a: Sending DeleteTablet for 3 replicas of tablet 200a5640a4694a06b52e459663473df6
I20260924 04:47:39.373087 10895 tablet_service.cc:1514] Processing DeleteTablet for tablet 200a5640a4694a06b52e459663473df6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33342
I20260924 04:47:39.373131 11029 tablet_service.cc:1514] Processing DeleteTablet for tablet 200a5640a4694a06b52e459663473df6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48920
I20260924 04:47:39.373168 10761 tablet_service.cc:1514] Processing DeleteTablet for tablet 200a5640a4694a06b52e459663473df6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:47:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60596
I20260924 04:47:39.373600 11138 tablet_replica.cc:330] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1: stopping tablet replica
I20260924 04:47:39.373919 11138 raft_consensus.cc:2229] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:47:39.374255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 10698
I20260924 04:47:39.374270 11138 raft_consensus.cc:2258] T 200a5640a4694a06b52e459663473df6 P 8f1fe0379e5a40b4924b5d96d26e32d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:39.374600 11137 tablet_replica.cc:330] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7: stopping tablet replica
I20260924 04:47:39.374986 11137 raft_consensus.cc:2229] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:39.375268 11137 raft_consensus.cc:2258] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:39.376662 11137 ts_tablet_manager.cc:1905] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:39.378777 11137 ts_tablet_manager.cc:1918] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:47:39.378859 11137 log.cc:1196] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/200a5640a4694a06b52e459663473df6
I20260924 04:47:39.379117 11136 tablet_replica.cc:330] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0: stopping tablet replica
I20260924 04:47:39.379165 11137 ts_tablet_manager.cc:1939] T 200a5640a4694a06b52e459663473df6 P d863b7eb46ef4490b602c5f6af5001a7: Deleting consensus metadata
I20260924 04:47:39.379448 11136 raft_consensus.cc:2229] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:47:39.379789 11136 raft_consensus.cc:2258] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:47:39.380585 10622 catalog_manager.cc:4841] TS d863b7eb46ef4490b602c5f6af5001a7 (127.0.98.3:39921): tablet 200a5640a4694a06b52e459663473df6 (table test-table [id=cf31c30733a846c3844523b5acb73db0]) successfully deleted
W20260924 04:47:39.380676 10623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37085 (error 108)
I20260924 04:47:39.381291 11136 ts_tablet_manager.cc:1905] T 200a5640a4694a06b52e459663473df6 P eded914d5391420391a5e67e9af178e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:47:39.381479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 10832
I20260924 04:47:39.381665 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.381319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37085 (local address 127.0.0.1:60624)
0924 04:47:39.381510 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:39.381584 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:47:39.382107 10623 catalog_manager.cc:4584] TS 8f1fe0379e5a40b4924b5d96d26e32d1 (127.0.98.1:37085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 200a5640a4694a06b52e459663473df6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37085: connect: Connection refused (error 111)
I20260924 04:47:39.386868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 10966
I20260924 04:47:39.387652 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:47:39.387098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45077 (local address 127.0.0.1:33354)
0924 04:47:39.387509 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 04:47:39.387570 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335}
I20260924 04:47:39.391592 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 10604
2026-09-24T04:47:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2348 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:47:39 dist-test-slave-hvcl krb5kdc[11142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:47:39 dist-test-slave-hvcl krb5kdc[11142](info): set up 2 sockets
Sep 24 04:47:39 dist-test-slave-hvcl krb5kdc[11142](info): commencing operation
krb5kdc: starting...
W20260924 04:47:40.351534 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:47:40 dist-test-slave-hvcl krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:47:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:47:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 04:47:40.517217 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskI0qsqU/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/postgres -l logfile start
2026-09-24 04:47:43.010 UTC [11172] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 04:47:43.010 UTC [11172] LOG: listening on IPv4 address "127.0.98.20", port 54155
2026-09-24 04:47:43.014 UTC [11172] LOG: listening on Unix socket "/tmp/.s.PGSQL.54155"
2026-09-24 04:47:43.041 UTC [11175] LOG: database system was shut down at 2026-09-24 04:47:41 UTC
2026-09-24 04:47:43.044 UTC [11172] LOG: database system is ready to accept connections
I20260924 04:47:43.915414 392 mini_postgres.cc:96] Postgres bound to 54155
2026-09-24 04:47:43.921 UTC [11183] FATAL: database "slave" does not exist
127.0.98.20:54155 - accepting connections
I20260924 04:47:43.921756 392 mini_ranger.cc:161] Starting Ranger...
I20260924 04:47:43.936089 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 04:47:44.433187 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 04:47:44.433274 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-admin
2026-09-24 04:47:44,713 [I] DB FLAVOR :POSTGRES
2026-09-24 04:47:44,713 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:47:44,713 [I] Checking connection..
2026-09-24 04:47:44,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 04:47:44,945 [I] Checking connection passed.
2026-09-24 04:47:44,945 [I] --------- Verifying version history table ---------
2026-09-24 04:47:44,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:47:45,204 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:47:45,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:47:45,433 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:47:45,434 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 04:47:45,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 04:47:45.689 UTC [11247] WARNING: there is no transaction in progress
2026-09-24 04:47:45,695 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 04:47:45,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:47:45,955 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 04:47:45,955 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:47:45,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 04:47:46,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 04:47:46,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hvcl.c.gcp-upstream.internal','N') ;"
2026-09-24 04:47:46,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:47:46,941 [I] Table x_portal_user does not exist in database ranger
2026-09-24 04:47:46,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:47:47,170 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 04:47:47,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:47:47,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:47:47,667 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 04:47:47,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 04:47:48.649 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.665 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.681 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.696 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.713 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.732 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.793 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.800 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.815 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.827 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.838 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.853 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.863 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.869 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.876 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:48.883 UTC [11355] WARNING: there is no transaction in progress
2026-09-24 04:47:49,546 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 04:47:49,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 04:47:49,781 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 04:47:49,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:47:50,041 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 04:47:50,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:47:50,283 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 04:47:50,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:47:50,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:47:50,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hvcl.c.gcp-upstream.internal';"
2026-09-24 04:47:51,020 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 04:47:51,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 04:47:51,278 [I] DB_PATCHES have already been applied
I20260924 04:47:51.283754 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskI0qsqU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 04:47:51.283838 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 04:47:51.287025 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:57655
Sep 24, 2026 4:47:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:47:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:47:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:47:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:47:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:47:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:47:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 04:47:52.302384 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.381s user 0.000s sys 0.010s
Sep 24, 2026 4:48:11 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:48:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 24, 2026 4:48:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:48:11 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:48:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:48:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:48:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:48:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:48:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:48:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:48:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:48:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:48:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:48:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:48:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:48:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:48:11 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:48:12 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 04:48:12.729007 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 04:48:12 dist-test-slave-hvcl krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225292, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 04:48:12.817471 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 04:48:12.827431 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 04:48:13.733371 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 04:48:13.733453 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-kms
I20260924 04:48:13.733549 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54155
2026-09-24 04:48:13,906 [I] DB FLAVOR :POSTGRES
2026-09-24 04:48:13,906 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:48:13,906 [I] Checking connection
2026-09-24 04:48:13,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 04:48:14,186 [I] connection success
2026-09-24 04:48:14,186 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 04:48:14,186 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 04:48:14,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 04:48:14,487 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 04:48:14,488 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:48:14,488 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 04:48:14,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54155/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 04:48:14,777 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 04:48:15.198438 392 mini_ranger_kms.cc:310] Created kms service
I20260924 04:48:15.352882 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 04:48:15.408773 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 04:48:15 dist-test-slave-hvcl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225292, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:48:15.822304 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 04:48:15.822472 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 04:48:15.822499 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 04:48:15.826236 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11535
I20260924 04:48:15.826344 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:39067
Sep 24, 2026 4:48:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:48:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:48:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:48:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:48:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:48:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 04:48:17.907051 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.090s user 0.009s sys 0.147s
I20260924 04:48:17.907191 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 04:48:17.907298 392 mini_ranger_kms.cc:401] 127.0.98.20:39067/kms/v1/keys
Sep 24 04:48:18 dist-test-slave-hvcl krb5kdc[11142](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790225298, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 04:48:18 dist-test-slave-hvcl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225298, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:48:18 dist-test-slave-hvcl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225298, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:48:21 dist-test-slave-hvcl krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:48:21.245771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43295
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43295
--ranger_config_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39067/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:48:21.359110 11586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:48:21.359396 11586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:48:21.359436 11586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:48:21.363612 11586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:48:21.363682 11586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:48:21.363701 11586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:48:21.363716 11586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:48:21.363730 11586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:48:21.369365 11586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:39067/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43295
--ranger_config_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:48:21.370451 11586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:48:21.371543 11586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:48:21.379978 11591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:48:21.379979 11592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:48:21.380149 11594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:48:21.381031 11586 server_base.cc:1034] running on GCE node
I20260924 04:48:21.381798 11586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:48:21.383340 11586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:48:21.384538 11586 hybrid_clock.cc:648] HybridClock initialized: now 1790225301384517 us; error 31 us; skew 500 ppm
Sep 24 04:48:21 dist-test-slave-hvcl krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:48:21.387912 11586 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:48:21.389045 11586 webserver.cc:460] Webserver started at http://127.0.98.62:46281/ using document root <none> and password file <none>
I20260924 04:48:21.389724 11586 fs_manager.cc:359] Metadata directory not provided
I20260924 04:48:21.389799 11586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:48:21.390038 11586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:48:21 dist-test-slave-hvcl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:48:21.446661 11586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "5dcebc20a212463a84458baf79d2817c"
format_stamp: "Formatted at 2026-09-24 04:48:21 on dist-test-slave-hvcl"
server_key: "ee900f6acc7492680b6730b1e5c72aa9"
server_key_iv: "671604dae2ccb70df7862f8c1951a8a7"
server_key_version: "kuduclusterkey@0"
I20260924 04:48:21.447438 11586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "5dcebc20a212463a84458baf79d2817c"
format_stamp: "Formatted at 2026-09-24 04:48:21 on dist-test-slave-hvcl"
server_key: "ee900f6acc7492680b6730b1e5c72aa9"
server_key_iv: "671604dae2ccb70df7862f8c1951a8a7"
server_key_version: "kuduclusterkey@0"
I20260924 04:48:21.452395 11586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 04:48:21.470679 11601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:48:21.472394 11586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:48:21.472574 11586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "5dcebc20a212463a84458baf79d2817c"
format_stamp: "Formatted at 2026-09-24 04:48:21 on dist-test-slave-hvcl"
server_key: "ee900f6acc7492680b6730b1e5c72aa9"
server_key_iv: "671604dae2ccb70df7862f8c1951a8a7"
server_key_version: "kuduclusterkey@0"
I20260924 04:48:21.472721 11586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:48:21.521008 11586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:48:21.523732 11586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:48:21.523891 11586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:48:21.531045 11586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43295
I20260924 04:48:21.531090 11653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43295 every 8 connection(s)
I20260924 04:48:21.532220 11586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:48:21.533617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 11586
I20260924 04:48:21.533742 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:48:21.534060 392 external_mini_cluster.cc:1428] Setting key c4ba2540e65eb842214d1a9bcfed0083
I20260924 04:48:21.536677 11654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:48:21.543007 11654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c: Bootstrap starting.
Sep 24 04:48:21 dist-test-slave-hvcl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:48:21.545265 11654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c: Neither blocks nor log segments found. Creating new log.
I20260924 04:48:21.546320 11654 log.cc:824] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:48:21.548192 11654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c: No bootstrap required, opened a new log
I20260924 04:48:21.548719 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:21.535381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.0.98.62:43295)
0924 04:48:21.535793 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 04:48:21.535812 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:48:21.536704 (+ 892us) server_negotiation.cc:407] Connection header received
0924 04:48:21.537428 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:48:21.537445 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:48:21.537775 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:48:21.538062 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:48:21.538949 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:21.540044 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:21.541217 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:21.541702 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:21.541829 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:21.544500 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:48:21.544505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:48:21.544518 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:48:21.544547 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:48:21.545981 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:48:21.546425 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:48:21.546430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:48:21.546436 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:48:21.546554 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:48:21.546795 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:48:21.546798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:48:21.546800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:48:21.547230 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:48:21.547401 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 04:48:21.547671 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 04:48:21.547935 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":131,"threads_started":1}
I20260924 04:48:21.551160 11654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcebc20a212463a84458baf79d2817c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43295 } }
I20260924 04:48:21.551375 11654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:48:21.551429 11654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcebc20a212463a84458baf79d2817c, State: Initialized, Role: FOLLOWER
I20260924 04:48:21.551819 11654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcebc20a212463a84458baf79d2817c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43295 } }
I20260924 04:48:21.551975 11654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:48:21.552034 11654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:48:21.552135 11654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:48:21.552999 11654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcebc20a212463a84458baf79d2817c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43295 } }
I20260924 04:48:21.553365 11654 leader_election.cc:302] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dcebc20a212463a84458baf79d2817c; no voters:
I20260924 04:48:21.553710 11654 leader_election.cc:288] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:48:21.553766 11659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:48:21.553993 11659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [term 1 LEADER]: Becoming Leader. State: Replica: 5dcebc20a212463a84458baf79d2817c, State: Running, Role: LEADER
I20260924 04:48:21.554329 11659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcebc20a212463a84458baf79d2817c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43295 } }
I20260924 04:48:21.554884 11654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:48:21.555985 11661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dcebc20a212463a84458baf79d2817c. Latest consensus state: current_term: 1 leader_uuid: "5dcebc20a212463a84458baf79d2817c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcebc20a212463a84458baf79d2817c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43295 } } }
I20260924 04:48:21.556015 11660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dcebc20a212463a84458baf79d2817c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcebc20a212463a84458baf79d2817c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43295 } } }
I20260924 04:48:21.556195 11661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [sys.catalog]: This master's current role is: LEADER
I20260924 04:48:21.556212 11660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c [sys.catalog]: This master's current role is: LEADER
I20260924 04:48:21.557431 11669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:48:21.557655 11654 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:48:21.562569 11669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:48:21.569340 11669 catalog_manager.cc:1347] Generated new cluster ID: eb0e3c3d8b63440f9a497c09c407057f
I20260924 04:48:21.569409 11669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:48:21.575663 11669 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:48:21.576782 11669 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:48:21.585745 11669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c: Generated new TSK 0
I20260924 04:48:21.586439 11669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:48:22.291034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43295
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:48:22.433614 11699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:48:22.434023 11699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:48:22.434098 11699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:48:22.440548 11699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:48:22.440658 11699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:48:22.440749 11699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:48:22.449045 11699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:39067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:48:22.450430 11699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:48:22.451792 11699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:48:22.464867 11707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:48:22.465165 11709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:48:22.464854 11706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:48:22.465478 11699 server_base.cc:1034] running on GCE node
I20260924 04:48:22.466068 11699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:48:22.466872 11699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:48:22.468056 11699 hybrid_clock.cc:648] HybridClock initialized: now 1790225302468015 us; error 55 us; skew 500 ppm
Sep 24 04:48:22 dist-test-slave-hvcl krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:48:22.470960 11699 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:48:22.472128 11699 webserver.cc:460] Webserver started at http://127.0.98.1:37867/ using document root <none> and password file <none>
I20260924 04:48:22.472836 11699 fs_manager.cc:359] Metadata directory not provided
I20260924 04:48:22.472915 11699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:48:22.473197 11699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:48:22 dist-test-slave-hvcl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:48:22.489851 11699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "c5cd7c8d15fd4fd88ee8880210426811"
format_stamp: "Formatted at 2026-09-24 04:48:22 on dist-test-slave-hvcl"
server_key: "dd8130c037d1b4781362161366f98d0e"
server_key_iv: "bcc2793cbbb55d8dec7f8a3ea03cd6ed"
server_key_version: "kuduclusterkey@0"
I20260924 04:48:22.490643 11699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c5cd7c8d15fd4fd88ee8880210426811"
format_stamp: "Formatted at 2026-09-24 04:48:22 on dist-test-slave-hvcl"
server_key: "dd8130c037d1b4781362161366f98d0e"
server_key_iv: "bcc2793cbbb55d8dec7f8a3ea03cd6ed"
server_key_version: "kuduclusterkey@0"
I20260924 04:48:22.507042 11699 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.001s
I20260924 04:48:22.526345 11718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:48:22.528689 11699 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:48:22.528839 11699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "c5cd7c8d15fd4fd88ee8880210426811"
format_stamp: "Formatted at 2026-09-24 04:48:22 on dist-test-slave-hvcl"
server_key: "dd8130c037d1b4781362161366f98d0e"
server_key_iv: "bcc2793cbbb55d8dec7f8a3ea03cd6ed"
server_key_version: "kuduclusterkey@0"
I20260924 04:48:22.528983 11699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:48:22.573297 11699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:48:22.577198 11699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:48:22.577415 11699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:48:22.578190 11699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:48:22.579308 11699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:48:22.579375 11699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:48:22.579428 11699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:48:22.579507 11699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:48:22.591568 11699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43523
I20260924 04:48:22.591614 11831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43523 every 8 connection(s)
I20260924 04:48:22.592955 11699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:48:22.596190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 11699
I20260924 04:48:22.596334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:48:22.596668 392 external_mini_cluster.cc:1428] Setting key f7ab1aea1dfb9e5239483c394cd3a724
Sep 24 04:48:22 dist-test-slave-hvcl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:48:22.617390 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:22.595618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43295 (local address 127.0.98.1:42051)
0924 04:48:22.597404 (+ 1786us) negotiation.cc:107] Waiting for socket to connect
0924 04:48:22.597461 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:48:22.600393 (+ 2932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:48:22.603704 (+ 3311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:48:22.603716 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:48:22.604091 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:48:22.604852 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:22.604874 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:22.606064 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:22.606068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:22.607608 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:22.607620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:22.608825 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:22.608830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:22.609044 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:22.609776 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:48:22.609798 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:48:22.611528 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:48:22.613825 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:48:22.613832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:48:22.613853 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:48:22.614231 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:48:22.614532 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:48:22.614536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:48:22.614538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:48:22.614734 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:48:22.615279 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:48:22.615287 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:48:22.615663 (+ 376us) client_negotiation.cc:769] Sending connection context
0924 04:48:22.615876 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 04:48:22.616245 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1554,"thread_start_us":139,"threads_started":1}
I20260924 04:48:22.618368 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:22.595217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42051 (local address 127.0.98.62:43295)
0924 04:48:22.603199 (+ 7982us) server_negotiation.cc:206] Beginning negotiation
0924 04:48:22.603204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:48:22.603223 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:48:22.603271 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:48:22.603274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:48:22.603329 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:48:22.603434 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:48:22.605036 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:22.605909 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:22.607762 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:22.608068 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:22.608179 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:22.611786 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:48:22.611791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:48:22.611794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:48:22.611827 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:48:22.613695 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:48:22.614352 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:48:22.614356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:48:22.614358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:48:22.614453 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:48:22.614845 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:48:22.614848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:48:22.614850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:48:22.615130 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:48:22.615226 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:48:22.618041 (+ 2815us) server_negotiation.cc:299] Negotiation successful
0924 04:48:22.618192 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7886,"thread_start_us":81,"threads_started":1}
I20260924 04:48:22.619771 11832 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43295
I20260924 04:48:22.620029 11832 heartbeater.cc:464] Registering TS with master...
I20260924 04:48:22.620656 11832 heartbeater.cc:511] Master 127.0.98.62:43295 requested a full tablet report, sending...
Sep 24 04:48:22 dist-test-slave-hvcl krb5kdc[11142](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790225302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:48:22.622541 11618 ts_manager.cc:194] Registered new tserver with Master: c5cd7c8d15fd4fd88ee8880210426811 (127.0.98.1:43523)
I20260924 04:48:22.624143 11618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:48:22.638376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43295
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:48:22.786393 11838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:48:22.786880 11838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:48:22.787037 11838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:48:22.793260 11838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:48:22.793357 11838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:48:22.793452 11838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:48:22.801635 11838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:39067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:48:22.803170 11838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:48:22.804567 11838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:48:22.817857 11844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:48:22.817801 11845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:48:22.818130 11838 server_base.cc:1034] running on GCE node
W20260924 04:48:22.818215 11847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:48:22.818943 11838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:48:22.819829 11838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:48:22.821082 11838 hybrid_clock.cc:648] HybridClock initialized: now 1790225302821047 us; error 55 us; skew 500 ppm
Sep 24 04:48:22 dist-test-slave-hvcl krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:48:22.824878 11838 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:48:22.826128 11838 webserver.cc:460] Webserver started at http://127.0.98.2:35613/ using document root <none> and password file <none>
I20260924 04:48:22.826900 11838 fs_manager.cc:359] Metadata directory not provided
I20260924 04:48:22.827025 11838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:48:22.827370 11838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:48:22 dist-test-slave-hvcl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:48:22.845057 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "f4d0da295d3b48bfaed7b5ac2074d81b"
format_stamp: "Formatted at 2026-09-24 04:48:22 on dist-test-slave-hvcl"
server_key: "1a2b0970a204527db1acbb966767809e"
server_key_iv: "e5c068f91d649e05f105773cef31c6fa"
server_key_version: "kuduclusterkey@0"
I20260924 04:48:22.846174 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f4d0da295d3b48bfaed7b5ac2074d81b"
format_stamp: "Formatted at 2026-09-24 04:48:22 on dist-test-slave-hvcl"
server_key: "1a2b0970a204527db1acbb966767809e"
server_key_iv: "e5c068f91d649e05f105773cef31c6fa"
server_key_version: "kuduclusterkey@0"
I20260924 04:48:22.852499 11838 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 04:48:22.875211 11854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:48:22.877275 11838 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260924 04:48:22.877640 11838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "f4d0da295d3b48bfaed7b5ac2074d81b"
format_stamp: "Formatted at 2026-09-24 04:48:22 on dist-test-slave-hvcl"
server_key: "1a2b0970a204527db1acbb966767809e"
server_key_iv: "e5c068f91d649e05f105773cef31c6fa"
server_key_version: "kuduclusterkey@0"
I20260924 04:48:22.877889 11838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:48:22.909157 11838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:48:22.913019 11838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:48:22.913211 11838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:48:22.913904 11838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:48:22.914894 11838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:48:22.914953 11838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:48:22.915032 11838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:48:22.915062 11838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:48:22.927163 11838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37873
I20260924 04:48:22.928359 11838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:48:22.929929 11967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37873 every 8 connection(s)
I20260924 04:48:22.930410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 11838
I20260924 04:48:22.930552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:48:22.930840 392 external_mini_cluster.cc:1428] Setting key 3001235a882e78579b8691bc4d4daab4
Sep 24 04:48:22 dist-test-slave-hvcl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:48:22.948660 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:22.933428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43667 (local address 127.0.98.62:43295)
0924 04:48:22.933553 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:48:22.933558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:48:22.935079 (+ 1521us) server_negotiation.cc:407] Connection header received
0924 04:48:22.936324 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:48:22.936328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:48:22.936382 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:48:22.936432 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:48:22.938053 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:22.938921 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:22.941116 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:22.941445 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:22.941513 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:22.944541 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:48:22.944545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:48:22.944549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:48:22.944578 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:48:22.946251 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:48:22.946866 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:48:22.946870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:48:22.946872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:48:22.946954 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:48:22.947351 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:48:22.947355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:48:22.947357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:48:22.947581 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:48:22.947648 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:48:22.948341 (+ 693us) server_negotiation.cc:299] Negotiation successful
0924 04:48:22.948476 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:48:22.950570 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:22.934322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43295 (local address 127.0.98.2:43667)
0924 04:48:22.934895 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 04:48:22.934957 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 04:48:22.936070 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:48:22.936875 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:48:22.936884 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:48:22.937226 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:48:22.937889 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:22.937907 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:22.939098 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:22.939101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:22.940978 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:22.940989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:22.941722 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:22.941725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:22.941920 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:22.942599 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:48:22.942620 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:48:22.944338 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:48:22.946389 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:48:22.946395 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:48:22.946411 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:48:22.946755 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:48:22.947081 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:48:22.947084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:48:22.947086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:48:22.947252 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:48:22.947725 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:48:22.947731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:48:22.948059 (+ 328us) client_negotiation.cc:769] Sending connection context
0924 04:48:22.948998 (+ 939us) client_negotiation.cc:240] Negotiation successful
0924 04:48:22.949368 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":156,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:48:22.952047 11968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43295
I20260924 04:48:22.952293 11968 heartbeater.cc:464] Registering TS with master...
I20260924 04:48:22.952869 11968 heartbeater.cc:511] Master 127.0.98.62:43295 requested a full tablet report, sending...
I20260924 04:48:22.953974 11618 ts_manager.cc:194] Registered new tserver with Master: f4d0da295d3b48bfaed7b5ac2074d81b (127.0.98.2:37873)
I20260924 04:48:22.954730 11618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:48:22.972322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43295
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:48:23 dist-test-slave-hvcl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 04:48:23.137609 11973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:48:23.138023 11973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:48:23.138100 11973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:48:23.143982 11973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:48:23.144065 11973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:48:23.144151 11973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:48:23.152323 11973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:39067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:48:23.153786 11973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:48:23.155236 11973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:48:23.169219 11973 server_base.cc:1034] running on GCE node
W20260924 04:48:23.171218 11981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:48:23.171339 11978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:48:23.171227 11979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:48:23.172032 11973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:48:23.172914 11973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:48:23.174083 11973 hybrid_clock.cc:648] HybridClock initialized: now 1790225303174013 us; error 90 us; skew 500 ppm
Sep 24 04:48:23 dist-test-slave-hvcl krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:48:23.177809 11973 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:48:23.179097 11973 webserver.cc:460] Webserver started at http://127.0.98.3:40887/ using document root <none> and password file <none>
I20260924 04:48:23.179883 11973 fs_manager.cc:359] Metadata directory not provided
I20260924 04:48:23.180022 11973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:48:23.180300 11973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:48:23 dist-test-slave-hvcl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:48:23.196519 11973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "183b5ea46c0e418da9bf1207328be585"
format_stamp: "Formatted at 2026-09-24 04:48:23 on dist-test-slave-hvcl"
server_key: "1e0d7626e2e66c6c13aa46497bf83298"
server_key_iv: "850127543ac7bb43e00bef388972479e"
server_key_version: "kuduclusterkey@0"
I20260924 04:48:23.197356 11973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "183b5ea46c0e418da9bf1207328be585"
format_stamp: "Formatted at 2026-09-24 04:48:23 on dist-test-slave-hvcl"
server_key: "1e0d7626e2e66c6c13aa46497bf83298"
server_key_iv: "850127543ac7bb43e00bef388972479e"
server_key_version: "kuduclusterkey@0"
I20260924 04:48:23.212934 11973 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.002s sys 0.004s
I20260924 04:48:23.241489 11988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:48:23.243441 11973 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.000s
I20260924 04:48:23.243615 11973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "183b5ea46c0e418da9bf1207328be585"
format_stamp: "Formatted at 2026-09-24 04:48:23 on dist-test-slave-hvcl"
server_key: "1e0d7626e2e66c6c13aa46497bf83298"
server_key_iv: "850127543ac7bb43e00bef388972479e"
server_key_version: "kuduclusterkey@0"
I20260924 04:48:23.243778 11973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:48:23.284020 11973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:48:23.288131 11973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:48:23.288347 11973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:48:23.289095 11973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:48:23.290187 11973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:48:23.290266 11973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:48:23.290326 11973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:48:23.290357 11973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:48:23.300907 11973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41119
I20260924 04:48:23.302237 11973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:48:23.305670 12101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41119 every 8 connection(s)
I20260924 04:48:23.309712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 11973
I20260924 04:48:23.309845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:48:23.310184 392 external_mini_cluster.cc:1428] Setting key 34275c0cc8cc464639806c6351d218b2
Sep 24 04:48:23 dist-test-slave-hvcl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:48:23.323616 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.308352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48679 (local address 127.0.98.62:43295)
0924 04:48:23.308506 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:48:23.308510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:48:23.309771 (+ 1261us) server_negotiation.cc:407] Connection header received
0924 04:48:23.311189 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:48:23.311194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:48:23.311264 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:48:23.311344 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:48:23.313243 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.314097 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.315918 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.316268 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.316342 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:23.319338 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:48:23.319343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:48:23.319345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:48:23.319374 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:48:23.321003 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:48:23.321603 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:48:23.321607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:48:23.321609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:48:23.321713 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:48:23.322100 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:48:23.322103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:48:23.322105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:48:23.322337 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:48:23.322434 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:48:23.323310 (+ 876us) server_negotiation.cc:299] Negotiation successful
0924 04:48:23.323444 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:48:23.325145 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.308911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43295 (local address 127.0.98.3:48679)
0924 04:48:23.309563 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0924 04:48:23.309639 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 04:48:23.310820 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:48:23.311832 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:48:23.311843 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:48:23.312326 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:48:23.313074 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:23.313095 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.314259 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.314262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:23.315776 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:23.315787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.316554 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.316557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:23.316773 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:23.317446 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:48:23.317468 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:48:23.319144 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:48:23.321132 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:48:23.321139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:48:23.321158 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:48:23.321479 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:48:23.321808 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:48:23.321811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:48:23.321813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:48:23.321988 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:48:23.322458 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:48:23.322465 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:48:23.322811 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 04:48:23.323773 (+ 962us) client_negotiation.cc:240] Negotiation successful
0924 04:48:23.324120 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":197,"threads_started":1}
I20260924 04:48:23.326279 12102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43295
I20260924 04:48:23.326550 12102 heartbeater.cc:464] Registering TS with master...
I20260924 04:48:23.327189 12102 heartbeater.cc:511] Master 127.0.98.62:43295 requested a full tablet report, sending...
I20260924 04:48:23.328369 11618 ts_manager.cc:194] Registered new tserver with Master: 183b5ea46c0e418da9bf1207328be585 (127.0.98.3:41119)
I20260924 04:48:23.329149 11618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48679
I20260924 04:48:23.336966 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:48:23.348914 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.339350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33006 (local address 127.0.98.62:43295)
0924 04:48:23.339467 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:48:23.339471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:48:23.339762 (+ 291us) server_negotiation.cc:407] Connection header received
0924 04:48:23.339893 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:48:23.339897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:48:23.339944 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:48:23.340024 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:48:23.340900 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.341670 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.342862 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.343214 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.343323 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:23.345211 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:48:23.345215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:48:23.345218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:48:23.345249 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:48:23.347068 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:48:23.347539 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:48:23.347542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:48:23.347544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:48:23.347647 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:48:23.347933 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:48:23.347937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:48:23.347938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:48:23.348175 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:48:23.348307 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:48:23.348621 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 04:48:23.348748 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:48:23.352483 11618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:48:23.357174 11618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:48:23.376509 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.367949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37873 (local address 127.0.0.1:51372)
0924 04:48:23.370082 (+ 2133us) negotiation.cc:107] Waiting for socket to connect
0924 04:48:23.370132 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:48:23.370269 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:48:23.371426 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:48:23.371433 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:48:23.371456 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:48:23.371584 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:23.371590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.373098 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.373101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:23.374531 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:23.374543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.375950 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.375954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:23.376082 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:23.376122 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 04:48:23.376229 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 04:48:23.376351 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1950,"thread_start_us":113,"threads_started":1}
I20260924 04:48:23.378064 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.369578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.0.98.2:37873)
0924 04:48:23.370772 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0924 04:48:23.370783 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:48:23.370805 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:48:23.370876 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:48:23.370882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:48:23.371117 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:48:23.371263 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:48:23.371727 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.372970 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.374687 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.375612 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.375721 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:23.375880 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 04:48:23.376998 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0924 04:48:23.377882 (+ 884us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":111,"threads_started":1}
I20260924 04:48:23.381702 11890 tablet_service.cc:1467] Processing CreateTablet for tablet c88558cb7fb045a4a45afea5c78456bb (DEFAULT_TABLE table=test-table [id=0f6138ac07c24f2f9d4c670967bf8149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:48:23.382879 11890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88558cb7fb045a4a45afea5c78456bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:48:23.383421 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.369046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43523 (local address 127.0.0.1:49852)
0924 04:48:23.374786 (+ 5740us) negotiation.cc:107] Waiting for socket to connect
0924 04:48:23.374799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:48:23.374875 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:48:23.375903 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:48:23.375906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:48:23.375925 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:48:23.376020 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:23.376026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.379600 (+ 3574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.379604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:23.380746 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:23.380757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.382849 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.382853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:23.383065 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:23.383090 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:48:23.383157 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:48:23.383228 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5674,"mutex_wait_us":57,"thread_start_us":75,"threads_started":1}
I20260924 04:48:23.383853 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.369220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.0.98.1:43523)
0924 04:48:23.375219 (+ 5999us) server_negotiation.cc:206] Beginning negotiation
0924 04:48:23.375228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:48:23.375255 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:48:23.375337 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:48:23.375343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:48:23.375591 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:48:23.375758 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:48:23.376153 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.379450 (+ 3297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.380886 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.382528 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.382645 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:23.382790 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:48:23.383614 (+ 824us) server_negotiation.cc:299] Negotiation successful
0924 04:48:23.383721 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5870,"thread_start_us":80,"threads_started":1}
I20260924 04:48:23.385181 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.368748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41119 (local address 127.0.0.1:45354)
0924 04:48:23.370472 (+ 1724us) negotiation.cc:107] Waiting for socket to connect
0924 04:48:23.370484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:48:23.370546 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:48:23.380251 (+ 9705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:48:23.380254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:48:23.380275 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:48:23.380382 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:23.380388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.381861 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.381865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:23.382948 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:23.382959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.384794 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.384798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:23.384897 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:23.384921 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:48:23.384984 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:48:23.385053 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1668,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260924 04:48:23.385262 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.369561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.0.98.3:41119)
0924 04:48:23.379773 (+ 10212us) server_negotiation.cc:206] Beginning negotiation
0924 04:48:23.379785 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:48:23.379810 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:48:23.379883 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:48:23.379888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:48:23.380090 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:48:23.380529 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:48:23.380677 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.381733 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.383227 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.384675 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.384778 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:23.384972 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 04:48:23.385025 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 04:48:23.385110 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10098,"thread_start_us":110,"threads_started":1}
I20260924 04:48:23.386988 11748 tablet_service.cc:1467] Processing CreateTablet for tablet c88558cb7fb045a4a45afea5c78456bb (DEFAULT_TABLE table=test-table [id=0f6138ac07c24f2f9d4c670967bf8149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:48:23.388121 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88558cb7fb045a4a45afea5c78456bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:48:23.388108 12034 tablet_service.cc:1467] Processing CreateTablet for tablet c88558cb7fb045a4a45afea5c78456bb (DEFAULT_TABLE table=test-table [id=0f6138ac07c24f2f9d4c670967bf8149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:48:23.389062 12034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88558cb7fb045a4a45afea5c78456bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:48:23.396927 12122 tablet_bootstrap.cc:492] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585: Bootstrap starting.
I20260924 04:48:23.399940 12120 tablet_bootstrap.cc:492] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b: Bootstrap starting.
I20260924 04:48:23.400434 12121 tablet_bootstrap.cc:492] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811: Bootstrap starting.
I20260924 04:48:23.402313 12120 tablet_bootstrap.cc:654] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b: Neither blocks nor log segments found. Creating new log.
I20260924 04:48:23.402645 12121 tablet_bootstrap.cc:654] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811: Neither blocks nor log segments found. Creating new log.
I20260924 04:48:23.403553 12121 log.cc:824] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811: Log is configured to *not* fsync() on all Append() calls
I20260924 04:48:23.403692 12120 log.cc:824] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:48:23.404531 12122 tablet_bootstrap.cc:654] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585: Neither blocks nor log segments found. Creating new log.
I20260924 04:48:23.405357 12122 log.cc:824] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585: Log is configured to *not* fsync() on all Append() calls
I20260924 04:48:23.407709 12120 tablet_bootstrap.cc:492] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b: No bootstrap required, opened a new log
I20260924 04:48:23.407912 12120 ts_tablet_manager.cc:1397] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 04:48:23.409287 12122 tablet_bootstrap.cc:492] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585: No bootstrap required, opened a new log
I20260924 04:48:23.409483 12122 ts_tablet_manager.cc:1397] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260924 04:48:23.412473 12120 raft_consensus.cc:348] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d0da295d3b48bfaed7b5ac2074d81b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37873 } } peers { permanent_uuid: "183b5ea46c0e418da9bf1207328be585" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } peers { permanent_uuid: "c5cd7c8d15fd4fd88ee8880210426811" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43523 } }
I20260924 04:48:23.412793 12120 raft_consensus.cc:374] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:48:23.412859 12120 raft_consensus.cc:729] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d0da295d3b48bfaed7b5ac2074d81b, State: Initialized, Role: FOLLOWER
I20260924 04:48:23.413378 12120 consensus_queue.cc:260] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d0da295d3b48bfaed7b5ac2074d81b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37873 } } peers { permanent_uuid: "183b5ea46c0e418da9bf1207328be585" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } peers { permanent_uuid: "c5cd7c8d15fd4fd88ee8880210426811" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43523 } }
I20260924 04:48:23.413275 12122 raft_consensus.cc:348] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d0da295d3b48bfaed7b5ac2074d81b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37873 } } peers { permanent_uuid: "183b5ea46c0e418da9bf1207328be585" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } peers { permanent_uuid: "c5cd7c8d15fd4fd88ee8880210426811" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43523 } }
I20260924 04:48:23.413519 12122 raft_consensus.cc:374] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:48:23.413566 12122 raft_consensus.cc:729] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183b5ea46c0e418da9bf1207328be585, State: Initialized, Role: FOLLOWER
I20260924 04:48:23.413975 12122 consensus_queue.cc:260] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d0da295d3b48bfaed7b5ac2074d81b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37873 } } peers { permanent_uuid: "183b5ea46c0e418da9bf1207328be585" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } peers { permanent_uuid: "c5cd7c8d15fd4fd88ee8880210426811" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43523 } }
I20260924 04:48:23.414516 12120 ts_tablet_manager.cc:1428] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 04:48:23.414912 12122 ts_tablet_manager.cc:1428] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:48:23.414927 11968 heartbeater.cc:503] Master 127.0.98.62:43295 was elected leader, sending a full tablet report...
I20260924 04:48:23.415264 12102 heartbeater.cc:503] Master 127.0.98.62:43295 was elected leader, sending a full tablet report...
I20260924 04:48:23.422960 12121 tablet_bootstrap.cc:492] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811: No bootstrap required, opened a new log
I20260924 04:48:23.423233 12121 ts_tablet_manager.cc:1397] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20260924 04:48:23.427712 12121 raft_consensus.cc:348] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d0da295d3b48bfaed7b5ac2074d81b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37873 } } peers { permanent_uuid: "183b5ea46c0e418da9bf1207328be585" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } peers { permanent_uuid: "c5cd7c8d15fd4fd88ee8880210426811" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43523 } }
I20260924 04:48:23.427987 12121 raft_consensus.cc:374] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:48:23.428025 12121 raft_consensus.cc:729] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cd7c8d15fd4fd88ee8880210426811, State: Initialized, Role: FOLLOWER
I20260924 04:48:23.428560 12121 consensus_queue.cc:260] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d0da295d3b48bfaed7b5ac2074d81b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37873 } } peers { permanent_uuid: "183b5ea46c0e418da9bf1207328be585" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } peers { permanent_uuid: "c5cd7c8d15fd4fd88ee8880210426811" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43523 } }
I20260924 04:48:23.429824 12121 ts_tablet_manager.cc:1428] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:48:23.430236 11832 heartbeater.cc:503] Master 127.0.98.62:43295 was elected leader, sending a full tablet report...
W20260924 04:48:23.431656 11969 tablet.cc:2369] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:48:23 dist-test-slave-hvcl krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:48:23.537431 12127 raft_consensus.cc:482] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:48:23.537773 12127 raft_consensus.cc:504] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d0da295d3b48bfaed7b5ac2074d81b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37873 } } peers { permanent_uuid: "183b5ea46c0e418da9bf1207328be585" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } peers { permanent_uuid: "c5cd7c8d15fd4fd88ee8880210426811" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43523 } }
I20260924 04:48:23.539541 12127 leader_election.cc:288] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4d0da295d3b48bfaed7b5ac2074d81b (127.0.98.2:37873), c5cd7c8d15fd4fd88ee8880210426811 (127.0.98.1:43523)
I20260924 04:48:23.545415 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.539925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37873 (local address 127.0.98.3:45099)
0924 04:48:23.540073 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 04:48:23.540090 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:48:23.540195 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:48:23.540972 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:48:23.540975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:48:23.540992 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:48:23.541101 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:23.541107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.542640 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.542643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:23.543979 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:23.543991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.544991 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.544994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:23.545104 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:23.545125 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:48:23.545201 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 04:48:23.545273 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":19}
I20260924 04:48:23.545869 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.540570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45099 (local address 127.0.98.2:37873)
0924 04:48:23.540678 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:48:23.540683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:48:23.540697 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:48:23.540749 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:48:23.540751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:48:23.540793 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:48:23.540861 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:48:23.541227 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.542054 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.544135 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.544762 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.544865 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:23.544930 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:48:23.545677 (+ 747us) server_negotiation.cc:299] Negotiation successful
0924 04:48:23.545759 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:48:23.546424 11909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88558cb7fb045a4a45afea5c78456bb" candidate_uuid: "183b5ea46c0e418da9bf1207328be585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d0da295d3b48bfaed7b5ac2074d81b" is_pre_election: true
I20260924 04:48:23.546752 11909 raft_consensus.cc:2454] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 183b5ea46c0e418da9bf1207328be585 in term 0.
I20260924 04:48:23.547283 11992 leader_election.cc:302] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183b5ea46c0e418da9bf1207328be585, f4d0da295d3b48bfaed7b5ac2074d81b; no voters:
I20260924 04:48:23.547592 12127 raft_consensus.cc:2781] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:48:23.547665 12127 raft_consensus.cc:482] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:48:23.547704 12127 raft_consensus.cc:3037] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:48:23.548945 12127 raft_consensus.cc:504] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d0da295d3b48bfaed7b5ac2074d81b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37873 } } peers { permanent_uuid: "183b5ea46c0e418da9bf1207328be585" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } peers { permanent_uuid: "c5cd7c8d15fd4fd88ee8880210426811" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43523 } }
I20260924 04:48:23.549077 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.542247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44337 (local address 127.0.98.1:43523)
0924 04:48:23.542365 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:48:23.542368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:48:23.542381 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:48:23.542427 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:48:23.542430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:48:23.542470 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:48:23.542538 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:48:23.544801 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.545567 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.547670 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.548272 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.548353 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:23.548416 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:48:23.548849 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 04:48:23.548932 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:48:23.549516 12127 leader_election.cc:288] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [CANDIDATE]: Term 1 election: Requested vote from peers f4d0da295d3b48bfaed7b5ac2074d81b (127.0.98.2:37873), c5cd7c8d15fd4fd88ee8880210426811 (127.0.98.1:43523)
I20260924 04:48:23.549731 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.540159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43523 (local address 127.0.98.3:44337)
0924 04:48:23.542045 (+ 1886us) negotiation.cc:107] Waiting for socket to connect
0924 04:48:23.542058 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:48:23.542153 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:48:23.542577 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:48:23.542580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:48:23.542599 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:48:23.544658 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:23.544666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.546121 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.546124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:23.547533 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:48:23.547544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.548612 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.548615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:48:23.548714 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:23.548737 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:48:23.549445 (+ 708us) client_negotiation.cc:240] Negotiation successful
0924 04:48:23.549522 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1822,"mutex_wait_us":1,"thread_start_us":129,"threads_started":1}
I20260924 04:48:23.550052 11909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88558cb7fb045a4a45afea5c78456bb" candidate_uuid: "183b5ea46c0e418da9bf1207328be585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d0da295d3b48bfaed7b5ac2074d81b"
I20260924 04:48:23.550189 11909 raft_consensus.cc:3037] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:48:23.550549 11780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88558cb7fb045a4a45afea5c78456bb" candidate_uuid: "183b5ea46c0e418da9bf1207328be585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cd7c8d15fd4fd88ee8880210426811" is_pre_election: true
I20260924 04:48:23.550823 11780 raft_consensus.cc:2454] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 183b5ea46c0e418da9bf1207328be585 in term 0.
I20260924 04:48:23.551182 11779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88558cb7fb045a4a45afea5c78456bb" candidate_uuid: "183b5ea46c0e418da9bf1207328be585" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cd7c8d15fd4fd88ee8880210426811"
I20260924 04:48:23.551280 11779 raft_consensus.cc:3037] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:48:23.551390 11909 raft_consensus.cc:2454] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 183b5ea46c0e418da9bf1207328be585 in term 1.
I20260924 04:48:23.552491 11779 raft_consensus.cc:2454] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 183b5ea46c0e418da9bf1207328be585 in term 1.
I20260924 04:48:23.553248 11992 leader_election.cc:302] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 183b5ea46c0e418da9bf1207328be585, c5cd7c8d15fd4fd88ee8880210426811; no voters:
I20260924 04:48:23.553548 12127 raft_consensus.cc:2781] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:48:23.553781 12127 raft_consensus.cc:686] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [term 1 LEADER]: Becoming Leader. State: Replica: 183b5ea46c0e418da9bf1207328be585, State: Running, Role: LEADER
I20260924 04:48:23.554169 12127 consensus_queue.cc:237] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d0da295d3b48bfaed7b5ac2074d81b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37873 } } peers { permanent_uuid: "183b5ea46c0e418da9bf1207328be585" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } } peers { permanent_uuid: "c5cd7c8d15fd4fd88ee8880210426811" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43523 } }
W20260924 04:48:23.556697 12103 tablet.cc:2369] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:48:23.560030 11618 catalog_manager.cc:5495] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 reported cstate change: term changed from 0 to 1, leader changed from <none> to 183b5ea46c0e418da9bf1207328be585 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "183b5ea46c0e418da9bf1207328be585" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d0da295d3b48bfaed7b5ac2074d81b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "183b5ea46c0e418da9bf1207328be585" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5cd7c8d15fd4fd88ee8880210426811" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43523 } health_report { overall_health: UNKNOWN } } }
W20260924 04:48:23.594951 11833 tablet.cc:2369] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:48:23.625550 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.620042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45370 (local address 127.0.98.3:41119)
0924 04:48:23.620500 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 04:48:23.620505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:48:23.620521 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:48:23.620615 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:48:23.620619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:48:23.620695 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:48:23.620816 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:48:23.621194 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.621999 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.623253 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:48:23.623559 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:48:23.623675 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:48:23.624135 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:48:23.624747 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:48:23.624870 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:48:23.625328 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 04:48:23.625393 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369}
I20260924 04:48:23.634728 11779 raft_consensus.cc:1261] T c88558cb7fb045a4a45afea5c78456bb P c5cd7c8d15fd4fd88ee8880210426811 [term 1 FOLLOWER]: Refusing update from remote peer 183b5ea46c0e418da9bf1207328be585: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:48:23.635797 11909 raft_consensus.cc:1261] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b [term 1 FOLLOWER]: Refusing update from remote peer 183b5ea46c0e418da9bf1207328be585: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:48:23.637048 12132 consensus_queue.cc:1035] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cd7c8d15fd4fd88ee8880210426811" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:48:23.637579 12132 consensus_queue.cc:1035] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d0da295d3b48bfaed7b5ac2074d81b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:48:23.668051 12138 mvcc.cc:205] Tried to move back new op lower bound from 7332762843675938816 to 7332762843368722432. Current Snapshot: MvccSnapshot[applied={T|T < 7332762843675938816}]
I20260924 04:48:23.668303 12136 mvcc.cc:205] Tried to move back new op lower bound from 7332762843675938816 to 7332762843368722432. Current Snapshot: MvccSnapshot[applied={T|T < 7332762843675938816}]
I20260924 04:48:23.682857 11618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:48:23.683071 11618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:48:23.685714 11618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5dcebc20a212463a84458baf79d2817c: Sending DeleteTablet for 3 replicas of tablet c88558cb7fb045a4a45afea5c78456bb
I20260924 04:48:23.686642 11890 tablet_service.cc:1514] Processing DeleteTablet for tablet c88558cb7fb045a4a45afea5c78456bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:48:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51372
I20260924 04:48:23.686913 12034 tablet_service.cc:1514] Processing DeleteTablet for tablet c88558cb7fb045a4a45afea5c78456bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:48:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45354
I20260924 04:48:23.687376 11748 tablet_service.cc:1514] Processing DeleteTablet for tablet c88558cb7fb045a4a45afea5c78456bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:48:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49852
I20260924 04:48:23.688571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 11699
I20260924 04:48:23.694015 12145 tablet_replica.cc:330] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585: stopping tablet replica
I20260924 04:48:23.694355 12145 raft_consensus.cc:2229] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:48:23.694789 12145 raft_consensus.cc:2258] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:48:23.699079 12145 ts_tablet_manager.cc:1905] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:48:23.699759 12144 tablet_replica.cc:330] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b: stopping tablet replica
I20260924 04:48:23.700093 12144 raft_consensus.cc:2229] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:48:23.700448 12144 raft_consensus.cc:2258] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:48:23.702049 12144 ts_tablet_manager.cc:1905] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:48:23.704607 12144 ts_tablet_manager.cc:1918] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:48:23.704718 12144 log.cc:1196] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/c88558cb7fb045a4a45afea5c78456bb
I20260924 04:48:23.705006 12144 ts_tablet_manager.cc:1939] T c88558cb7fb045a4a45afea5c78456bb P f4d0da295d3b48bfaed7b5ac2074d81b: Deleting consensus metadata
I20260924 04:48:23.706067 12145 ts_tablet_manager.cc:1918] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:48:23.706329 11605 catalog_manager.cc:4841] TS f4d0da295d3b48bfaed7b5ac2074d81b (127.0.98.2:37873): tablet c88558cb7fb045a4a45afea5c78456bb (table test-table [id=0f6138ac07c24f2f9d4c670967bf8149]) successfully deleted
W20260924 04:48:23.706689 11605 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43523 (error 108)
I20260924 04:48:23.706866 12145 log.cc:1196] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/c88558cb7fb045a4a45afea5c78456bb
I20260924 04:48:23.707357 12145 ts_tablet_manager.cc:1939] T c88558cb7fb045a4a45afea5c78456bb P 183b5ea46c0e418da9bf1207328be585: Deleting consensus metadata
I20260924 04:48:23.707572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 11838
I20260924 04:48:23.713717 11605 catalog_manager.cc:4841] TS 183b5ea46c0e418da9bf1207328be585 (127.0.98.3:41119): tablet c88558cb7fb045a4a45afea5c78456bb (table test-table [id=0f6138ac07c24f2f9d4c670967bf8149]) successfully deleted
I20260924 04:48:23.715675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 11973
I20260924 04:48:23.724084 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:48:23.723739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43523 (local address 127.0.0.1:49868)
0924 04:48:23.723891 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 04:48:23.723982 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:48:23.724342 11605 catalog_manager.cc:4584] TS c5cd7c8d15fd4fd88ee8880210426811 (127.0.98.1:43523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c88558cb7fb045a4a45afea5c78456bb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43523: connect: Connection refused (error 111)
I20260924 04:48:23.735785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 11586
I20260924 04:48:23.741837 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 04:48:24.076912 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 04:48:24.077031 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 04:48:24.411974 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 04:48:24.412 UTC [11172] LOG: received smart shutdown request
2026-09-24 04:48:24.416 UTC [11172] LOG: background worker "logical replication launcher" (PID 11181) exited with exit code 1
2026-09-24 04:48:24.416 UTC [11176] LOG: shutting down
2026-09-24 04:48:24.589 UTC [11172] LOG: database system is shut down
2026-09-24T04:48:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45290 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:48:24 dist-test-slave-hvcl krb5kdc[12156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:48:24 dist-test-slave-hvcl krb5kdc[12156](info): set up 2 sockets
Sep 24 04:48:24 dist-test-slave-hvcl krb5kdc[12156](info): commencing operation
krb5kdc: starting...
W20260924 04:48:25.634905 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:48:25 dist-test-slave-hvcl krb5kdc[12156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:48:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:48:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 04:48:25.797673 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskI0qsqU/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/postgres -l logfile start
2026-09-24 04:48:28.349 UTC [12186] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 04:48:28.349 UTC [12186] LOG: listening on IPv4 address "127.0.98.20", port 55069
2026-09-24 04:48:28.353 UTC [12186] LOG: listening on Unix socket "/tmp/.s.PGSQL.55069"
2026-09-24 04:48:28.372 UTC [12189] LOG: database system was shut down at 2026-09-24 04:48:26 UTC
2026-09-24 04:48:28.376 UTC [12186] LOG: database system is ready to accept connections
I20260924 04:48:29.250763 392 mini_postgres.cc:96] Postgres bound to 55069
2026-09-24 04:48:29.256 UTC [12197] FATAL: database "slave" does not exist
127.0.98.20:55069 - accepting connections
I20260924 04:48:29.256897 392 mini_ranger.cc:161] Starting Ranger...
I20260924 04:48:29.271086 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 04:48:29.765865 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 04:48:29.765935 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-admin
2026-09-24 04:48:30,037 [I] DB FLAVOR :POSTGRES
2026-09-24 04:48:30,037 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:48:30,037 [I] Checking connection..
2026-09-24 04:48:30,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 04:48:30,297 [I] Checking connection passed.
2026-09-24 04:48:30,298 [I] --------- Verifying version history table ---------
2026-09-24 04:48:30,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:48:30,549 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:48:30,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:48:30,785 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:48:30,786 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 04:48:30,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 04:48:31.022 UTC [12261] WARNING: there is no transaction in progress
2026-09-24 04:48:31,028 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 04:48:31,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:48:31,257 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 04:48:31,257 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:48:31,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 04:48:31,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 04:48:31,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hvcl.c.gcp-upstream.internal','N') ;"
2026-09-24 04:48:31,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:48:32,251 [I] Table x_portal_user does not exist in database ranger
2026-09-24 04:48:32,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:48:32,481 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 04:48:32,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:48:32,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:48:32,975 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 04:48:32,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 04:48:33.929 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:33.945 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:33.962 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:33.979 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:33.995 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:34.016 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:34.078 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:34.085 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:34.100 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:34.110 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:34.121 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:34.135 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:34.145 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:34.152 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:34.159 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:34.166 UTC [12369] WARNING: there is no transaction in progress
2026-09-24 04:48:34,817 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 04:48:34,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 04:48:35,071 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 04:48:35,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:48:35,321 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 04:48:35,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:48:35,574 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 04:48:35,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:48:35,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:48:36,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hvcl.c.gcp-upstream.internal';"
2026-09-24 04:48:36,286 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 04:48:36,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 04:48:36,541 [I] DB_PATCHES have already been applied
I20260924 04:48:36.546446 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskI0qsqU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 04:48:36.546514 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 04:48:36.549642 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:43049
Sep 24, 2026 4:48:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:48:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:48:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:48:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:48:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:48:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:48:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 04:48:37.523581 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.267s user 0.000s sys 0.009s
Sep 24, 2026 4:48:57 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:48:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 4:48:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:48:57 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:48:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:48:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:48:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:48:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:48:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:48:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:48:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:48:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:48:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:48:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:48:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:48:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:48:57 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:48:58 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 04:48:58.901346 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 04:48:58 dist-test-slave-hvcl krb5kdc[12156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225338, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 04:48:58.985832 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 04:48:58.995503 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 04:48:59.914240 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 04:48:59.914314 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-kms
I20260924 04:48:59.914325 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:55069
2026-09-24 04:49:00,068 [I] DB FLAVOR :POSTGRES
2026-09-24 04:49:00,069 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:49:00,069 [I] Checking connection
2026-09-24 04:49:00,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 04:49:00,308 [I] connection success
2026-09-24 04:49:00,308 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 04:49:00,308 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 04:49:00,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 04:49:00,545 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 04:49:00,546 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:49:00,546 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 04:49:00,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:55069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 04:49:00,795 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 04:49:01.275378 392 mini_ranger_kms.cc:310] Created kms service
I20260924 04:49:01.475607 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 04:49:01.517262 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 04:49:01 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225338, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:49:01.891443 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 04:49:01.891607 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskI0qsqU/build/debug/bin/postgresql.jar:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskI0qsqU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 04:49:01.891630 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 04:49:01.895584 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12549
I20260924 04:49:01.895673 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:47607
Sep 24, 2026 4:49:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:49:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:49:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:49:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:49:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:49:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 04:49:03.929961 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.944s user 0.021s sys 0.116s
I20260924 04:49:03.930084 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 04:49:03.930159 392 mini_ranger_kms.cc:401] 127.0.98.20:47607/kms/v1/keys
Sep 24 04:49:04 dist-test-slave-hvcl krb5kdc[12156](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790225344, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 04:49:04 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225344, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:49:04 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225344, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:49:07 dist-test-slave-hvcl krb5kdc[12156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:49:07.274672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38207
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38207
--ranger_config_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47607/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:07.402223 12600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:07.402525 12600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:07.402565 12600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:07.406781 12600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:49:07.406857 12600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:07.406874 12600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:07.406889 12600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:49:07.406903 12600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:49:07.412776 12600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:47607/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38207
--ranger_config_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:07.413899 12600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:07.414960 12600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:07.424098 12605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:07.424124 12608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:07.424187 12600 server_base.cc:1034] running on GCE node
W20260924 04:49:07.424100 12606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:07.424906 12600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:07.426086 12600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:07.427260 12600 hybrid_clock.cc:648] HybridClock initialized: now 1790225347427230 us; error 41 us; skew 500 ppm
Sep 24 04:49:07 dist-test-slave-hvcl krb5kdc[12156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:07.430075 12600 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:49:07.431100 12600 webserver.cc:460] Webserver started at http://127.0.98.62:45329/ using document root <none> and password file <none>
I20260924 04:49:07.431641 12600 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:07.431700 12600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:07.431897 12600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:49:07 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:49:07.478235 12600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "1a8a06076fa8454295a66d83a34ec5a7"
format_stamp: "Formatted at 2026-09-24 04:49:07 on dist-test-slave-hvcl"
server_key: "6521cecfa7fb733cf19dedbe670007fb"
server_key_iv: "0212c36c90becc57bf54751055ebee49"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:07.478811 12600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a8a06076fa8454295a66d83a34ec5a7"
format_stamp: "Formatted at 2026-09-24 04:49:07 on dist-test-slave-hvcl"
server_key: "6521cecfa7fb733cf19dedbe670007fb"
server_key_iv: "0212c36c90becc57bf54751055ebee49"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:07.482625 12600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.001s
I20260924 04:49:07.496522 12615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:07.497936 12600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:49:07.498068 12600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "1a8a06076fa8454295a66d83a34ec5a7"
format_stamp: "Formatted at 2026-09-24 04:49:07 on dist-test-slave-hvcl"
server_key: "6521cecfa7fb733cf19dedbe670007fb"
server_key_iv: "0212c36c90becc57bf54751055ebee49"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:07.498205 12600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:07.512125 12600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:07.515308 12600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:07.515487 12600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:07.522878 12600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38207
I20260924 04:49:07.522921 12667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38207 every 8 connection(s)
I20260924 04:49:07.523871 12600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:49:07.527410 12668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:07.532821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 12600
I20260924 04:49:07.532967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:49:07.533236 392 external_mini_cluster.cc:1428] Setting key 4f0be4e58dd15916dbb7c7944d2a2dd1
I20260924 04:49:07.536306 12668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7: Bootstrap starting.
I20260924 04:49:07.539276 12668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:07.540256 12668 log.cc:824] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:07.542645 12668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7: No bootstrap required, opened a new log
Sep 24 04:49:07 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:07.547600 12668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8a06076fa8454295a66d83a34ec5a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } }
I20260924 04:49:07.547933 12668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:07.548015 12668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8a06076fa8454295a66d83a34ec5a7, State: Initialized, Role: FOLLOWER
I20260924 04:49:07.548558 12668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8a06076fa8454295a66d83a34ec5a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } }
I20260924 04:49:07.548722 12668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:49:07.548784 12668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:49:07.548869 12668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:07.550082 12668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8a06076fa8454295a66d83a34ec5a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } }
I20260924 04:49:07.550406 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:07.536256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.0.98.62:38207)
0924 04:49:07.537258 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:07.537286 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:07.538470 (+ 1184us) server_negotiation.cc:407] Connection header received
0924 04:49:07.539462 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:07.539491 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:07.539931 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:07.540264 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:07.541147 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:07.542135 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:07.543057 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:07.543566 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:07.543709 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:07.545981 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:07.545987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:07.546001 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:07.546035 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:07.547526 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:07.547985 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:07.547991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:07.547997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:07.548160 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:07.548408 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:07.548411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:07.548413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:07.548850 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:07.549057 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:07.549350 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 04:49:07.549639 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":187,"threads_started":1}
I20260924 04:49:07.550516 12668 leader_election.cc:302] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a8a06076fa8454295a66d83a34ec5a7; no voters:
I20260924 04:49:07.550861 12668 leader_election.cc:288] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:49:07.550947 12673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:49:07.551167 12673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 1 LEADER]: Becoming Leader. State: Replica: 1a8a06076fa8454295a66d83a34ec5a7, State: Running, Role: LEADER
I20260924 04:49:07.551456 12673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8a06076fa8454295a66d83a34ec5a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } }
I20260924 04:49:07.552193 12668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:49:07.552627 12673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a8a06076fa8454295a66d83a34ec5a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8a06076fa8454295a66d83a34ec5a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } } }
I20260924 04:49:07.552726 12673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:07.552878 12673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a8a06076fa8454295a66d83a34ec5a7. Latest consensus state: current_term: 1 leader_uuid: "1a8a06076fa8454295a66d83a34ec5a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8a06076fa8454295a66d83a34ec5a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } } }
I20260924 04:49:07.552942 12673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:07.553613 12677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:49:07.556942 12677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:49:07.557300 12668 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:49:07.568487 12677 catalog_manager.cc:1347] Generated new cluster ID: 429fdb5209174afea3b96a0d27e70816
I20260924 04:49:07.568572 12677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:49:07.587170 12677 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:49:07.588613 12677 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:49:07.594951 12677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7: Generated new TSK 0
I20260924 04:49:07.595862 12677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:49:08.242265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38207
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:08.392005 12713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:08.392302 12713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:08.392365 12713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:08.396927 12713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:08.397075 12713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:08.397188 12713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:49:08.403149 12713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:47607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:08.404289 12713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:08.405351 12713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:08.415053 12721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:08.415165 12720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:08.415294 12713 server_base.cc:1034] running on GCE node
W20260924 04:49:08.415088 12723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:08.416127 12713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:08.417024 12713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:08.418241 12713 hybrid_clock.cc:648] HybridClock initialized: now 1790225348418181 us; error 79 us; skew 500 ppm
Sep 24 04:49:08 dist-test-slave-hvcl krb5kdc[12156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:08.422277 12713 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:49:08.423607 12713 webserver.cc:460] Webserver started at http://127.0.98.1:40807/ using document root <none> and password file <none>
I20260924 04:49:08.424398 12713 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:08.424501 12713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:08.424772 12713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:49:08 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:49:08.441866 12713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "a94fbf3bb10a4e3688522272b7406dfe"
format_stamp: "Formatted at 2026-09-24 04:49:08 on dist-test-slave-hvcl"
server_key: "9b74bfac1696ff8b29ec3e5786444c74"
server_key_iv: "7bb301d852e163c451b35c10787bc90a"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:08.442837 12713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a94fbf3bb10a4e3688522272b7406dfe"
format_stamp: "Formatted at 2026-09-24 04:49:08 on dist-test-slave-hvcl"
server_key: "9b74bfac1696ff8b29ec3e5786444c74"
server_key_iv: "7bb301d852e163c451b35c10787bc90a"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:08.448714 12713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 04:49:08.475431 12730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:08.477264 12713 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.000s
I20260924 04:49:08.477396 12713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "a94fbf3bb10a4e3688522272b7406dfe"
format_stamp: "Formatted at 2026-09-24 04:49:08 on dist-test-slave-hvcl"
server_key: "9b74bfac1696ff8b29ec3e5786444c74"
server_key_iv: "7bb301d852e163c451b35c10787bc90a"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:08.477541 12713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:08.499497 12713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:08.503562 12713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:08.503849 12713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:08.504635 12713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:08.505847 12713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:08.506000 12713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:08.506141 12713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:08.506215 12713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:08.516592 12713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41071
I20260924 04:49:08.517906 12713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:49:08.520740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 12713
I20260924 04:49:08.520857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:49:08.521199 392 external_mini_cluster.cc:1428] Setting key b15e95863cbcd5a103c6147dac6e665e
I20260924 04:49:08.523331 12845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41071 every 8 connection(s)
Sep 24 04:49:08 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:08.538208 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:08.521077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57923 (local address 127.0.98.62:38207)
0924 04:49:08.521426 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:08.521435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:08.525643 (+ 4208us) server_negotiation.cc:407] Connection header received
0924 04:49:08.525690 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:08.525693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:08.525745 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:08.525823 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:08.527362 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:08.528248 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:08.529942 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:08.530483 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:08.530560 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:08.533730 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:08.533735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:08.533737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:08.533767 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:08.535668 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:08.536338 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:08.536341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:08.536343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:08.536434 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:08.536838 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:08.536841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:08.536843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:08.537090 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:08.537171 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:08.537883 (+ 712us) server_negotiation.cc:299] Negotiation successful
0924 04:49:08.538032 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":101,"threads_started":1}
I20260924 04:49:08.540232 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:08.521535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38207 (local address 127.0.98.1:57923)
0924 04:49:08.524208 (+ 2673us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:08.524265 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:08.525324 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:08.526137 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:08.526147 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:08.526494 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:08.527199 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:08.527218 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:08.528405 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:08.528409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:08.529808 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:08.529819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:08.530778 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:08.530782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:08.531025 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:08.531707 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:08.531728 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:08.533516 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:08.535819 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:08.535826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:08.535845 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:08.536216 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:08.536537 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:08.536540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:08.536542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:08.536731 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:08.537238 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:08.537246 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:08.537605 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 04:49:08.538671 (+ 1066us) client_negotiation.cc:240] Negotiation successful
0924 04:49:08.539077 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2424,"thread_start_us":129,"threads_started":1}
I20260924 04:49:08.541918 12846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38207
I20260924 04:49:08.542176 12846 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:49:08.542766 12846 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
I20260924 04:49:08.544651 12632 ts_manager.cc:194] Registered new tserver with Master: a94fbf3bb10a4e3688522272b7406dfe (127.0.98.1:41071)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:49:08.546306 12632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:49:08.564664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38207
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:49:08 dist-test-slave-hvcl krb5kdc[12156](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790225348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 04:49:08.721695 12852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:08.722086 12852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:08.722167 12852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:08.728379 12852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:08.728470 12852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:08.728570 12852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:49:08.736764 12852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:47607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:08.738217 12852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:08.739610 12852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:08.752061 12861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:08.752161 12852 server_base.cc:1034] running on GCE node
W20260924 04:49:08.752285 12859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:08.752470 12858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:08.752908 12852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:08.753616 12852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:08.754825 12852 hybrid_clock.cc:648] HybridClock initialized: now 1790225348754794 us; error 40 us; skew 500 ppm
Sep 24 04:49:08 dist-test-slave-hvcl krb5kdc[12156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:08.757958 12852 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:49:08.759161 12852 webserver.cc:460] Webserver started at http://127.0.98.2:40007/ using document root <none> and password file <none>
I20260924 04:49:08.759939 12852 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:08.760082 12852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:08.760360 12852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:49:08 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:49:08.785066 12852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9"
format_stamp: "Formatted at 2026-09-24 04:49:08 on dist-test-slave-hvcl"
server_key: "6c86aa3883f9644a3fb897217e4b5036"
server_key_iv: "085a1491c07c5205782ab0dcb512b49f"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:08.785708 12852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9"
format_stamp: "Formatted at 2026-09-24 04:49:08 on dist-test-slave-hvcl"
server_key: "6c86aa3883f9644a3fb897217e4b5036"
server_key_iv: "085a1491c07c5205782ab0dcb512b49f"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:08.790352 12852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:49:08.815129 12868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:08.816691 12852 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260924 04:49:08.816838 12852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9"
format_stamp: "Formatted at 2026-09-24 04:49:08 on dist-test-slave-hvcl"
server_key: "6c86aa3883f9644a3fb897217e4b5036"
server_key_iv: "085a1491c07c5205782ab0dcb512b49f"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:08.816979 12852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:08.827117 12852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:08.830833 12852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:08.831073 12852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:08.831839 12852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:08.832990 12852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:08.833052 12852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:08.833108 12852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:08.833130 12852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:08.842917 12852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42359
I20260924 04:49:08.844322 12981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42359 every 8 connection(s)
I20260924 04:49:08.845785 12852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:49:08.855544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 12852
I20260924 04:49:08.855667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:49:08.855979 392 external_mini_cluster.cc:1428] Setting key 46ac8012a9d34e601592bd0b54617a1c
Sep 24 04:49:08 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:49:08.871657 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:08.847891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54899 (local address 127.0.98.62:38207)
0924 04:49:08.848031 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:08.848036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:08.850213 (+ 2177us) server_negotiation.cc:407] Connection header received
0924 04:49:08.851551 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:08.851555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:08.851642 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:08.851741 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:08.853171 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:08.854064 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:08.855814 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:08.856178 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:08.856264 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:08.863368 (+ 7104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:08.863373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:08.863376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:08.863415 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:08.865274 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:08.869078 (+ 3804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:08.869083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:08.869085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:08.869198 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:08.869714 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:08.869717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:08.869719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:08.869980 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:08.870073 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:08.871304 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0924 04:49:08.871467 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:49:08.873437 12984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:08.848787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38207 (local address 127.0.98.2:54899)
0924 04:49:08.850037 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:08.850096 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:08.851257 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:08.851933 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:08.851943 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:08.852293 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:08.853005 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:08.853026 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:08.854206 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:08.854210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:08.855670 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:08.855680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:08.856494 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:08.856498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:08.856718 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:08.857937 (+ 1219us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:08.857959 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:08.863081 (+ 5122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:08.868426 (+ 5345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:08.868436 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:08.868458 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:08.868913 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:08.869326 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:08.869329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:08.869331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:08.869587 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:08.870146 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:08.870155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:08.870561 (+ 406us) client_negotiation.cc:769] Sending connection context
0924 04:49:08.871905 (+ 1344us) client_negotiation.cc:240] Negotiation successful
0924 04:49:08.872297 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":148,"threads_started":1}
I20260924 04:49:08.874827 12982 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38207
I20260924 04:49:08.875108 12982 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:49:08.875726 12982 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
I20260924 04:49:08.876868 12632 ts_manager.cc:194] Registered new tserver with Master: 7033fc0ada4a4dcfa9038ef7dcb2c3b9 (127.0.98.2:42359)
I20260924 04:49:08.877636 12632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:49:08.894316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38207
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:49:08 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 04:49:09.055290 12987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:09.055687 12987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:09.055753 12987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:09.061904 12987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:09.062004 12987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:09.062093 12987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:49:09.070314 12987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:47607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:09.071755 12987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:09.073113 12987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:09.085404 12992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:09.085404 12995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:09.085400 12993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:09.086097 12987 server_base.cc:1034] running on GCE node
I20260924 04:49:09.086563 12987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:09.087416 12987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:09.088608 12987 hybrid_clock.cc:648] HybridClock initialized: now 1790225349088578 us; error 44 us; skew 500 ppm
Sep 24 04:49:09 dist-test-slave-hvcl krb5kdc[12156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:09.092408 12987 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:49:09.093540 12987 webserver.cc:460] Webserver started at http://127.0.98.3:46073/ using document root <none> and password file <none>
I20260924 04:49:09.094220 12987 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:09.094302 12987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:09.094533 12987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:49:09 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:49:09.113517 12987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "92d149a4bec34c0c9684812a9d290fa5"
format_stamp: "Formatted at 2026-09-24 04:49:09 on dist-test-slave-hvcl"
server_key: "f7b582705d828635ab3612a5cd1e63b2"
server_key_iv: "3db6114efe84e6c82cbff12fa4aa52ad"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:09.114622 12987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "92d149a4bec34c0c9684812a9d290fa5"
format_stamp: "Formatted at 2026-09-24 04:49:09 on dist-test-slave-hvcl"
server_key: "f7b582705d828635ab3612a5cd1e63b2"
server_key_iv: "3db6114efe84e6c82cbff12fa4aa52ad"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:09.120390 12987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 04:49:09.140163 13002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:09.142000 12987 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260924 04:49:09.142153 12987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "92d149a4bec34c0c9684812a9d290fa5"
format_stamp: "Formatted at 2026-09-24 04:49:09 on dist-test-slave-hvcl"
server_key: "f7b582705d828635ab3612a5cd1e63b2"
server_key_iv: "3db6114efe84e6c82cbff12fa4aa52ad"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:09.142305 12987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:09.161932 12987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:09.165417 12987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:09.165589 12987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:09.166270 12987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:09.167318 12987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:09.167374 12987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:09.167433 12987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:09.167457 12987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:09.177179 12987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33215
I20260924 04:49:09.178364 12987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:49:09.186884 13115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33215 every 8 connection(s)
I20260924 04:49:09.187736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 12987
I20260924 04:49:09.187850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:49:09.188160 392 external_mini_cluster.cc:1428] Setting key dd9fa85a77a8ac1f811c388fe7344998
Sep 24 04:49:09 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:09.198819 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:09.182285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53367 (local address 127.0.98.62:38207)
0924 04:49:09.182432 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:09.182436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:09.182453 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:49:09.183252 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:09.183256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:09.183318 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:09.183411 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:09.185093 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:09.185939 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:09.187619 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:09.187956 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:09.188030 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:09.190917 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:09.190921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:09.190924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:09.190956 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:09.195930 (+ 4974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:09.196724 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:09.196727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:09.196729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:09.196827 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:09.197335 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:09.197339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:09.197341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:09.197616 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:09.197722 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:09.198508 (+ 786us) server_negotiation.cc:299] Negotiation successful
0924 04:49:09.198654 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:49:09.200624 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:09.181134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38207 (local address 127.0.98.3:53367)
0924 04:49:09.181743 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:09.181804 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:09.182880 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:09.183746 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:09.183757 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:09.184194 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:09.184883 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:09.184901 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:09.186065 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:09.186069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:09.187467 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:09.187479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:09.188235 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:09.188239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:09.188454 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:09.189124 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:09.189144 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:09.190723 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:09.196124 (+ 5401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:09.196132 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:09.196155 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:09.196591 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:09.196948 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:09.196951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:09.196954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:09.197176 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:09.197783 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:09.197792 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:09.198175 (+ 383us) client_negotiation.cc:769] Sending connection context
0924 04:49:09.199040 (+ 865us) client_negotiation.cc:240] Negotiation successful
0924 04:49:09.199483 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":157,"threads_started":1}
I20260924 04:49:09.202167 13116 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38207
I20260924 04:49:09.202407 13116 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:09.203015 13116 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
I20260924 04:49:09.204129 12628 ts_manager.cc:194] Registered new tserver with Master: 92d149a4bec34c0c9684812a9d290fa5 (127.0.98.3:33215)
I20260924 04:49:09.204931 12628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53367
I20260924 04:49:09.214738 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:49:09.215570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 12713
I20260924 04:49:09.223063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 12852
I20260924 04:49:09.229374 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 12987
I20260924 04:49:09.235191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 12600
I20260924 04:49:09.244649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38207
--webserver_interface=127.0.98.62
--webserver_port=45329
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38207
--ranger_config_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47607/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:47607/kms with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:09.355034 13119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:09.355314 13119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:09.355352 13119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:09.359412 13119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:49:09.359485 13119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:09.359503 13119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:09.359519 13119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:49:09.359532 13119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:49:09.365097 13119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:47607/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38207
--ranger_config_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:09.366180 13119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:09.367238 13119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:09.375726 13125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:09.375946 13119 server_base.cc:1034] running on GCE node
W20260924 04:49:09.375731 13127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:09.375771 13124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:09.376657 13119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:09.377895 13119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:09.379045 13119 hybrid_clock.cc:648] HybridClock initialized: now 1790225349379029 us; error 29 us; skew 500 ppm
Sep 24 04:49:09 dist-test-slave-hvcl krb5kdc[12156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:09.381839 13119 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:49:09.382741 13119 webserver.cc:460] Webserver started at http://127.0.98.62:45329/ using document root <none> and password file <none>
I20260924 04:49:09.383297 13119 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:09.383385 13119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:49:09 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:49:09.435905 13119 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:49:09.438252 13135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:09.439646 13119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:49:09.439785 13119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "1a8a06076fa8454295a66d83a34ec5a7"
format_stamp: "Formatted at 2026-09-24 04:49:07 on dist-test-slave-hvcl"
server_key: "6521cecfa7fb733cf19dedbe670007fb"
server_key_iv: "0212c36c90becc57bf54751055ebee49"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:09.440340 13119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:09.473230 13119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:09.475935 13119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:09.476142 13119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:09.483075 13119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38207
I20260924 04:49:09.483072 13187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38207 every 8 connection(s)
I20260924 04:49:09.484169 13119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:49:09.488343 13188 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:49:09.489959 13188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7: Bootstrap starting.
I20260924 04:49:09.492213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 13119
I20260924 04:49:09.492681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41071
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38207
--webserver_port=40807
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:47607/kms with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:49:09.506028 13188 log.cc:824] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:09.512764 13188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:49:09.513140 13188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7: Bootstrap complete.
I20260924 04:49:09.517197 13188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8a06076fa8454295a66d83a34ec5a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } }
I20260924 04:49:09.517601 13188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8a06076fa8454295a66d83a34ec5a7, State: Initialized, Role: FOLLOWER
I20260924 04:49:09.518064 13188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8a06076fa8454295a66d83a34ec5a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } }
I20260924 04:49:09.518181 13188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:49:09.518247 13188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:49:09.518337 13188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:49:09.520257 13188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8a06076fa8454295a66d83a34ec5a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } }
I20260924 04:49:09.520582 13188 leader_election.cc:302] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a8a06076fa8454295a66d83a34ec5a7; no voters:
I20260924 04:49:09.520869 13188 leader_election.cc:288] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:49:09.520972 13192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:49:09.521199 13192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [term 2 LEADER]: Becoming Leader. State: Replica: 1a8a06076fa8454295a66d83a34ec5a7, State: Running, Role: LEADER
I20260924 04:49:09.521518 13192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8a06076fa8454295a66d83a34ec5a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } }
I20260924 04:49:09.522097 13188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:49:09.522473 13192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a8a06076fa8454295a66d83a34ec5a7. Latest consensus state: current_term: 2 leader_uuid: "1a8a06076fa8454295a66d83a34ec5a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8a06076fa8454295a66d83a34ec5a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } } }
I20260924 04:49:09.522579 13192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:09.522938 13193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1a8a06076fa8454295a66d83a34ec5a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8a06076fa8454295a66d83a34ec5a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } } }
I20260924 04:49:09.523096 13193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:09.523023 13196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:49:09.526666 13196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:49:09.527246 13188 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:49:09.527304 13196 catalog_manager.cc:1259] Loaded cluster ID: 429fdb5209174afea3b96a0d27e70816
I20260924 04:49:09.527465 13196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:49:09.529771 13196 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:49:09.530437 13196 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7: Loaded TSK: 0
I20260924 04:49:09.531656 13196 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:49:09.622905 13190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:09.623235 13190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:09.623303 13190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:09.628196 13190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:09.628274 13190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:09.628360 13190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:49:09.633924 13190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:47607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40807
--tserver_master_addrs=127.0.98.62:38207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:09.635151 13190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:09.636396 13190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:09.648628 13226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:09.648662 13228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:09.649120 13190 server_base.cc:1034] running on GCE node
W20260924 04:49:09.649412 13225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:09.649850 13190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:09.650535 13190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:09.651714 13190 hybrid_clock.cc:648] HybridClock initialized: now 1790225349651688 us; error 39 us; skew 500 ppm
Sep 24 04:49:09 dist-test-slave-hvcl krb5kdc[12156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:09.654337 13190 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:49:09.655387 13190 webserver.cc:460] Webserver started at http://127.0.98.1:40807/ using document root <none> and password file <none>
I20260924 04:49:09.655969 13190 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:09.656051 13190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:49:09 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:49:09.695096 13190 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.000s
I20260924 04:49:09.698424 13236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:09.699934 13190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:49:09.700075 13190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "a94fbf3bb10a4e3688522272b7406dfe"
format_stamp: "Formatted at 2026-09-24 04:49:08 on dist-test-slave-hvcl"
server_key: "9b74bfac1696ff8b29ec3e5786444c74"
server_key_iv: "7bb301d852e163c451b35c10787bc90a"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:09.700798 13190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:09.718961 13190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:09.722481 13190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:09.722677 13190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:09.723655 13190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:09.724838 13190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:09.724900 13190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:09.724956 13190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:09.724985 13190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:09.735651 13190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41071
I20260924 04:49:09.735684 13349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41071 every 8 connection(s)
I20260924 04:49:09.736675 13190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:49:09.740123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 13190
I20260924 04:49:09.740612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42359
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38207
--webserver_port=40007
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:47607/kms with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:49:09 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:09.762580 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:09.738947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38207 (local address 127.0.98.1:34069)
0924 04:49:09.743630 (+ 4683us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:09.743693 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:09.745095 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:09.747353 (+ 2258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:09.747363 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:09.747735 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:09.748445 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:09.748464 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:09.750448 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:09.750452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:09.751924 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:09.751938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:09.753319 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:09.753323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:09.753517 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:09.754233 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:09.754254 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:09.756138 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:09.758846 (+ 2708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:09.758853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:09.758870 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:09.759342 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:09.759814 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:09.759818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:09.759820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:09.760019 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:09.760682 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:09.760689 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:09.760972 (+ 283us) client_negotiation.cc:769] Sending connection context
0924 04:49:09.761153 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 04:49:09.761519 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4427,"thread_start_us":152,"threads_started":1}
I20260924 04:49:09.762569 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:09.738909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34069 (local address 127.0.98.62:38207)
0924 04:49:09.743298 (+ 4389us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:09.743321 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:09.743962 (+ 641us) server_negotiation.cc:407] Connection header received
0924 04:49:09.745766 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:09.745782 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:09.746294 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:09.746700 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:09.748666 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:09.750273 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:09.752094 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:09.752892 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:09.753075 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:09.756434 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:09.756442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:09.756463 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:09.756504 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:09.758675 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:09.759508 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:09.759515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:09.759522 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:09.759691 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:09.760147 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:09.760151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:09.760153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:09.760564 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:09.760692 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:09.761318 (+ 626us) server_negotiation.cc:299] Negotiation successful
0924 04:49:09.761593 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4189,"thread_start_us":119,"threads_started":1}
I20260924 04:49:09.765415 13350 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38207
I20260924 04:49:09.765672 13350 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:09.766177 13350 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
W20260924 04:49:09.768445 13350 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38207 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:49:09.768589 13350 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:09.768778 13350 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
I20260924 04:49:09.769695 13350 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:09.769887 13350 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
W20260924 04:49:09.770665 13350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:49:09.909695 13354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:09.910121 13354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:09.910208 13354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:09.916540 13354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:09.916643 13354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:09.916738 13354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:49:09.925184 13354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:47607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40007
--tserver_master_addrs=127.0.98.62:38207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:09.926851 13354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:09.928396 13354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:09.940231 13362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:09.940227 13359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:09.940232 13360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:09.940732 13354 server_base.cc:1034] running on GCE node
I20260924 04:49:09.941250 13354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:09.941890 13354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:09.943053 13354 hybrid_clock.cc:648] HybridClock initialized: now 1790225349943019 us; error 54 us; skew 500 ppm
Sep 24 04:49:09 dist-test-slave-hvcl krb5kdc[12156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:09.949294 13354 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:49:09.950636 13354 webserver.cc:460] Webserver started at http://127.0.98.2:40007/ using document root <none> and password file <none>
I20260924 04:49:09.951512 13354 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:09.951632 13354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:49:09 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:49:09.976549 13354 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260924 04:49:09.980124 13370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:09.981806 13354 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 04:49:09.981933 13354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9"
format_stamp: "Formatted at 2026-09-24 04:49:08 on dist-test-slave-hvcl"
server_key: "6c86aa3883f9644a3fb897217e4b5036"
server_key_iv: "085a1491c07c5205782ab0dcb512b49f"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:09.982514 13354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:10.002081 13354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:10.007561 13354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:10.007758 13354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:10.008491 13354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:10.009438 13354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:10.009490 13354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:10.009544 13354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:10.009572 13354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:10.020284 13354 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42359
I20260924 04:49:10.020359 13483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42359 every 8 connection(s)
I20260924 04:49:10.021399 13354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:49:10.029686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 13354
I20260924 04:49:10.030184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33215
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38207
--webserver_port=46073
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:47607/kms with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:49:10 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:10.038020 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:10.024369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60539 (local address 127.0.98.62:38207)
0924 04:49:10.024523 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:10.024527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:10.026184 (+ 1657us) server_negotiation.cc:407] Connection header received
0924 04:49:10.027557 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:10.027561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:10.027623 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:10.027677 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:10.029021 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:10.029664 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:10.031654 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:10.031913 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:10.031973 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:10.034434 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:10.034438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:10.034440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:10.034468 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:10.035950 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:10.036447 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:10.036450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:10.036452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:10.036534 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:10.036857 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:10.036860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:10.036862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:10.037059 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:10.037143 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:10.037691 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 04:49:10.037823 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:49:10.039108 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:10.024886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38207 (local address 127.0.98.2:60539)
0924 04:49:10.026004 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:10.026065 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:10.027279 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:10.028082 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:10.028090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:10.028367 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:10.028865 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:10.028879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:10.029901 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:10.029905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:10.031499 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:10.031512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:10.032147 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:10.032149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:10.032373 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:10.032960 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:10.032980 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:10.034258 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:10.036067 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:10.036074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:10.036092 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:10.036353 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:10.036615 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:10.036618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:10.036620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:10.036771 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:10.037157 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:10.037164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:10.037444 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 04:49:10.037636 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 04:49:10.037973 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":146,"threads_started":1}
I20260924 04:49:10.040692 13484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38207
I20260924 04:49:10.040944 13484 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:10.041455 13484 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
W20260924 04:49:10.042941 13484 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38207 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:49:10.043095 13484 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:10.043263 13484 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
I20260924 04:49:10.044142 13484 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:10.044312 13484 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
W20260924 04:49:10.045385 13484 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:49:10.174876 13487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:10.175302 13487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:10.175377 13487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:10.181555 13487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:10.181654 13487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:10.181741 13487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:49:10.189992 13487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:47607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46073
--tserver_master_addrs=127.0.98.62:38207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:10.191468 13487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:10.192869 13487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:10.205029 13494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:10.205981 13496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:10.205855 13493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:10.206377 13487 server_base.cc:1034] running on GCE node
I20260924 04:49:10.207037 13487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:10.207877 13487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:10.209070 13487 hybrid_clock.cc:648] HybridClock initialized: now 1790225350209042 us; error 47 us; skew 500 ppm
Sep 24 04:49:10 dist-test-slave-hvcl krb5kdc[12156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:10.213101 13487 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:49:10.214401 13487 webserver.cc:460] Webserver started at http://127.0.98.3:46073/ using document root <none> and password file <none>
I20260924 04:49:10.215219 13487 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:10.215333 13487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:49:10 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:49:10.233747 13487 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:49:10.239888 13504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:10.241739 13487 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260924 04:49:10.241904 13487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "92d149a4bec34c0c9684812a9d290fa5"
format_stamp: "Formatted at 2026-09-24 04:49:09 on dist-test-slave-hvcl"
server_key: "f7b582705d828635ab3612a5cd1e63b2"
server_key_iv: "3db6114efe84e6c82cbff12fa4aa52ad"
server_key_version: "kuduclusterkey@0"
I20260924 04:49:10.242710 13487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:10.263634 13487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:10.267146 13487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:10.267335 13487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:10.268011 13487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:10.268898 13487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:10.268956 13487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:10.269022 13487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:10.269052 13487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:10.279330 13487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33215
I20260924 04:49:10.280362 13487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:49:10.287271 13617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33215 every 8 connection(s)
I20260924 04:49:10.288196 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 13487
Sep 24 04:49:10 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:10.300820 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:10.287002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58737 (local address 127.0.98.62:38207)
0924 04:49:10.287143 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:10.287147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:10.288069 (+ 922us) server_negotiation.cc:407] Connection header received
0924 04:49:10.290716 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:10.290720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:10.290780 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:10.290870 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:10.292545 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:10.293214 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:10.294448 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:10.294763 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:10.294850 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:10.297230 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:10.297233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:10.297237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:10.297261 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:10.298690 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:10.299298 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:10.299302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:10.299304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:10.299383 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:10.299709 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:10.299712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:10.299714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:10.299886 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:10.299967 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:10.300512 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 04:49:10.300649 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:49:10.302186 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:10.285847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38207 (local address 127.0.98.3:58737)
0924 04:49:10.287831 (+ 1984us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:10.287896 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:10.290443 (+ 2547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:10.291239 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:10.291250 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:10.291618 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:10.292340 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:10.292377 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:10.293353 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:10.293356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:10.294328 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:10.294337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:10.294911 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:10.294915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:10.295138 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:10.295724 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:10.295742 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:10.297063 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:10.298815 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:10.298821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:10.298839 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:10.299209 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:10.299459 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:10.299462 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:10.299464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:10.299615 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:10.299977 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:10.299985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:10.300256 (+ 271us) client_negotiation.cc:769] Sending connection context
0924 04:49:10.301002 (+ 746us) client_negotiation.cc:240] Negotiation successful
0924 04:49:10.301348 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"thread_start_us":150,"threads_started":1}
I20260924 04:49:10.303872 13618 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38207
I20260924 04:49:10.304177 13618 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:10.304791 13618 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
I20260924 04:49:10.306353 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:10.288895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.0.98.62:38207)
0924 04:49:10.289171 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:10.289178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:10.294444 (+ 5266us) server_negotiation.cc:407] Connection header received
0924 04:49:10.294479 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:10.294481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:10.294535 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:10.294611 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:10.298299 (+ 3688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:10.299096 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:10.300343 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:10.300650 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:10.300743 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:10.302799 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:10.302802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:10.302804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:10.302826 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:10.304372 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:10.304813 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:10.304817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:10.304819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:10.304913 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:10.305165 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:10.305168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:10.305170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:10.305374 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:10.305557 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:10.305931 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 04:49:10.306058 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":105,"threads_started":1}
W20260924 04:49:10.306656 13618 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38207 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:49:10.306749 13618 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:10.306911 13618 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
I20260924 04:49:10.308100 13618 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:10.308264 13618 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
W20260924 04:49:10.308965 13618 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 04:49:10 dist-test-slave-hvcl krb5kdc[12156](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790225350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:10.770927 13350 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:10.771272 13350 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
I20260924 04:49:10.772461 13152 ts_manager.cc:194] Registered new tserver with Master: a94fbf3bb10a4e3688522272b7406dfe (127.0.98.1:41071)
I20260924 04:49:10.774026 13152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34069
Sep 24 04:49:11 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:49:11.045594 13484 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:11.045900 13484 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
I20260924 04:49:11.050112 13151 ts_manager.cc:194] Registered new tserver with Master: 7033fc0ada4a4dcfa9038ef7dcb2c3b9 (127.0.98.2:42359)
I20260924 04:49:11.051074 13151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60539
I20260924 04:49:11.309185 13618 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:11.309518 13618 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
I20260924 04:49:11.310283 13151 ts_manager.cc:194] Registered new tserver with Master: 92d149a4bec34c0c9684812a9d290fa5 (127.0.98.3:33215)
I20260924 04:49:11.311110 13151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58737
I20260924 04:49:11.319433 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:49:11 dist-test-slave-hvcl krb5kdc[12156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:49:11.332211 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:11.320962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.0.98.62:38207)
0924 04:49:11.321785 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:11.321790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:11.322071 (+ 281us) server_negotiation.cc:407] Connection header received
0924 04:49:11.322182 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:11.322185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:11.322274 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:11.322352 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:11.323193 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.323969 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.324946 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.325365 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.325470 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:11.326839 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:11.326843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:11.326846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:11.326873 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:11.329613 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:11.330163 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:11.330167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:11.330169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:11.330264 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:11.330580 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:11.330584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:11.330586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:11.330820 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:11.330960 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:11.331654 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 04:49:11.331787 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":88,"threads_started":1}
I20260924 04:49:11.336988 13151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:49:11.341071 13151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:49:11.407395 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:11.383918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41071 (local address 127.0.0.1:51100)
0924 04:49:11.386568 (+ 2650us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:11.394005 (+ 7437us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:11.394329 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:11.397182 (+ 2853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:11.397185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:11.397202 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:11.397288 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:11.397294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.401714 (+ 4420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.401717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:11.402831 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:11.402843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.406774 (+ 3931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.406778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:11.406913 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:11.406954 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 04:49:11.407089 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 04:49:11.407234 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2449,"thread_start_us":111,"threads_started":1}
I20260924 04:49:11.408402 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:11.385859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42359 (local address 127.0.0.1:36178)
0924 04:49:11.394519 (+ 8660us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:11.394530 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:11.394618 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:11.397385 (+ 2767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:11.397388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:11.397399 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:11.397470 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:11.397474 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.398910 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.398914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:11.400305 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:11.400315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.408015 (+ 7700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.408019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:11.408112 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:11.408135 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:49:11.408197 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:49:11.408271 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8589,"thread_start_us":97,"threads_started":1}
I20260924 04:49:11.409138 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:11.386058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36178 (local address 127.0.98.2:42359)
0924 04:49:11.386780 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:11.386788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:11.394717 (+ 7929us) server_negotiation.cc:407] Connection header received
0924 04:49:11.394812 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:11.394819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:11.395121 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:11.395272 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:11.397592 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.398780 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.404291 (+ 5511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.405228 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.405333 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:11.405498 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:11.408874 (+ 3376us) server_negotiation.cc:299] Negotiation successful
0924 04:49:11.409001 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":89,"threads_started":1}
I20260924 04:49:11.409647 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:11.384750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51100 (local address 127.0.98.1:41071)
0924 04:49:11.396642 (+ 11892us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:11.396649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:11.396670 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:49:11.396738 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:11.396742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:11.396910 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:11.397035 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:11.400460 (+ 3425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.401580 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.405577 (+ 3997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.406448 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.406565 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:11.406710 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:11.409385 (+ 2675us) server_negotiation.cc:299] Negotiation successful
0924 04:49:11.409501 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11773,"thread_start_us":95,"threads_started":1}
I20260924 04:49:11.413034 13418 tablet_service.cc:1467] Processing CreateTablet for tablet 041729545d3d4da9a02294e2c2f780a5 (DEFAULT_TABLE table=test-table [id=e2ca7e77231e4111a9e52dc7cbf7fcad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:11.414247 13418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041729545d3d4da9a02294e2c2f780a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:11.424640 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:11.383959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33215 (local address 127.0.0.1:60622)
0924 04:49:11.390986 (+ 7027us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:11.391051 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:11.394185 (+ 3134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:11.396196 (+ 2011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:11.396202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:11.396226 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:11.396367 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:11.396373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.404254 (+ 7881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.404258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:11.416508 (+ 12250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:11.416529 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.424176 (+ 7647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.424181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:11.424295 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:11.424328 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 04:49:11.424380 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 04:49:11.424467 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5142,"thread_start_us":77,"threads_started":1}
I20260924 04:49:11.425670 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:11.384751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.0.98.3:33215)
0924 04:49:11.395588 (+ 10837us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:11.395596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:11.395621 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:49:11.395694 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:11.395700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:11.395890 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:11.396021 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:11.403014 (+ 6993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.404128 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.423066 (+ 18938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.424000 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.425181 (+ 1181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:11.425359 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:11.425416 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 04:49:11.425518 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10709,"thread_start_us":90,"threads_started":1}
I20260924 04:49:11.427898 13653 tablet_bootstrap.cc:492] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9: Bootstrap starting.
I20260924 04:49:11.429188 13552 tablet_service.cc:1467] Processing CreateTablet for tablet 041729545d3d4da9a02294e2c2f780a5 (DEFAULT_TABLE table=test-table [id=e2ca7e77231e4111a9e52dc7cbf7fcad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:11.430341 13552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041729545d3d4da9a02294e2c2f780a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:11.430596 13653 tablet_bootstrap.cc:654] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:11.431535 13653 log.cc:824] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:11.433997 13653 tablet_bootstrap.cc:492] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9: No bootstrap required, opened a new log
I20260924 04:49:11.434310 13653 ts_tablet_manager.cc:1397] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:49:11.437371 13284 tablet_service.cc:1467] Processing CreateTablet for tablet 041729545d3d4da9a02294e2c2f780a5 (DEFAULT_TABLE table=test-table [id=e2ca7e77231e4111a9e52dc7cbf7fcad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:11.438581 13284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041729545d3d4da9a02294e2c2f780a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:11.438586 13653 raft_consensus.cc:348] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94fbf3bb10a4e3688522272b7406dfe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "92d149a4bec34c0c9684812a9d290fa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33215 } } peers { permanent_uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42359 } }
I20260924 04:49:11.439045 13653 raft_consensus.cc:374] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:11.439141 13653 raft_consensus.cc:729] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7033fc0ada4a4dcfa9038ef7dcb2c3b9, State: Initialized, Role: FOLLOWER
I20260924 04:49:11.439692 13653 consensus_queue.cc:260] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94fbf3bb10a4e3688522272b7406dfe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "92d149a4bec34c0c9684812a9d290fa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33215 } } peers { permanent_uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42359 } }
I20260924 04:49:11.440904 13653 ts_tablet_manager.cc:1428] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:49:11.441311 13484 heartbeater.cc:503] Master 127.0.98.62:38207 was elected leader, sending a full tablet report...
I20260924 04:49:11.445634 13656 tablet_bootstrap.cc:492] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5: Bootstrap starting.
I20260924 04:49:11.447751 13657 tablet_bootstrap.cc:492] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe: Bootstrap starting.
I20260924 04:49:11.449391 13656 tablet_bootstrap.cc:654] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:11.449844 13657 tablet_bootstrap.cc:654] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:11.450387 13656 log.cc:824] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:11.450574 13657 log.cc:824] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:11.453233 13656 tablet_bootstrap.cc:492] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5: No bootstrap required, opened a new log
I20260924 04:49:11.453426 13656 ts_tablet_manager.cc:1397] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260924 04:49:11.454402 13657 tablet_bootstrap.cc:492] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe: No bootstrap required, opened a new log
I20260924 04:49:11.454586 13657 ts_tablet_manager.cc:1397] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260924 04:49:11.457604 13656 raft_consensus.cc:348] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94fbf3bb10a4e3688522272b7406dfe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "92d149a4bec34c0c9684812a9d290fa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33215 } } peers { permanent_uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42359 } }
I20260924 04:49:11.457903 13656 raft_consensus.cc:374] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:11.457964 13656 raft_consensus.cc:729] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d149a4bec34c0c9684812a9d290fa5, State: Initialized, Role: FOLLOWER
I20260924 04:49:11.458494 13656 consensus_queue.cc:260] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94fbf3bb10a4e3688522272b7406dfe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "92d149a4bec34c0c9684812a9d290fa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33215 } } peers { permanent_uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42359 } }
I20260924 04:49:11.458912 13657 raft_consensus.cc:348] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94fbf3bb10a4e3688522272b7406dfe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "92d149a4bec34c0c9684812a9d290fa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33215 } } peers { permanent_uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42359 } }
I20260924 04:49:11.459250 13657 raft_consensus.cc:374] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:11.459333 13657 raft_consensus.cc:729] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94fbf3bb10a4e3688522272b7406dfe, State: Initialized, Role: FOLLOWER
I20260924 04:49:11.459645 13656 ts_tablet_manager.cc:1428] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:49:11.459860 13657 consensus_queue.cc:260] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94fbf3bb10a4e3688522272b7406dfe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "92d149a4bec34c0c9684812a9d290fa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33215 } } peers { permanent_uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42359 } }
I20260924 04:49:11.460005 13618 heartbeater.cc:503] Master 127.0.98.62:38207 was elected leader, sending a full tablet report...
I20260924 04:49:11.460919 13657 ts_tablet_manager.cc:1428] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260924 04:49:11.461290 13350 heartbeater.cc:503] Master 127.0.98.62:38207 was elected leader, sending a full tablet report...
W20260924 04:49:11.489117 13351 tablet.cc:2369] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:49:11.524722 13485 tablet.cc:2369] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:49:11.535944 13619 tablet.cc:2369] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:49:11.605532 13661 raft_consensus.cc:482] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:49:11.605737 13661 raft_consensus.cc:504] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94fbf3bb10a4e3688522272b7406dfe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "92d149a4bec34c0c9684812a9d290fa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33215 } } peers { permanent_uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42359 } }
I20260924 04:49:11.606961 13661 leader_election.cc:288] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92d149a4bec34c0c9684812a9d290fa5 (127.0.98.3:33215), 7033fc0ada4a4dcfa9038ef7dcb2c3b9 (127.0.98.2:42359)
I20260924 04:49:11.611740 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:11.607703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42359 (local address 127.0.98.1:55671)
0924 04:49:11.608103 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:11.608115 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:11.608219 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:11.608476 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:11.608478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:11.608497 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:11.608593 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:11.608598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.609552 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.609556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:11.610572 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:11.610579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.611246 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.611249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:11.611342 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:11.611397 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 04:49:11.611479 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:49:11.611556 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":14,"spinlock_wait_cycles":384,"thread_start_us":82,"threads_started":1}
I20260924 04:49:11.611742 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:11.607323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50207 (local address 127.0.98.3:33215)
0924 04:49:11.607532 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:11.607536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:11.607678 (+ 142us) server_negotiation.cc:407] Connection header received
0924 04:49:11.607918 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:11.607923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:11.607980 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:11.608034 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:11.608469 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.609049 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.610465 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.611132 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.611244 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:11.611310 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:11.611563 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 04:49:11.611643 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260924 04:49:11.611792 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:11.607818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55671 (local address 127.0.98.2:42359)
0924 04:49:11.607936 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:11.607941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:11.608206 (+ 265us) server_negotiation.cc:407] Connection header received
0924 04:49:11.608320 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:11.608323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:11.608376 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:11.608461 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:11.608696 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.609425 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.610674 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.611142 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.611233 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:11.611297 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:11.611585 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 04:49:11.611667 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:49:11.612342 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:11.607255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33215 (local address 127.0.98.1:50207)
0924 04:49:11.607578 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:11.607592 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:11.607762 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:11.608184 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:11.608186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:11.608220 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:11.608361 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:11.608367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.609144 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.609147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:11.610307 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:11.610317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.611356 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.611360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:11.611438 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:11.611456 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:49:11.612105 (+ 649us) client_negotiation.cc:240] Negotiation successful
0924 04:49:11.612179 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"spinlock_wait_cycles":25088,"thread_start_us":140,"threads_started":1}
I20260924 04:49:11.612648 13438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041729545d3d4da9a02294e2c2f780a5" candidate_uuid: "a94fbf3bb10a4e3688522272b7406dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9" is_pre_election: true
I20260924 04:49:11.612939 13572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041729545d3d4da9a02294e2c2f780a5" candidate_uuid: "a94fbf3bb10a4e3688522272b7406dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d149a4bec34c0c9684812a9d290fa5" is_pre_election: true
I20260924 04:49:11.612993 13438 raft_consensus.cc:2454] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a94fbf3bb10a4e3688522272b7406dfe in term 0.
I20260924 04:49:11.613243 13572 raft_consensus.cc:2454] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a94fbf3bb10a4e3688522272b7406dfe in term 0.
I20260924 04:49:11.613462 13237 leader_election.cc:302] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7033fc0ada4a4dcfa9038ef7dcb2c3b9, a94fbf3bb10a4e3688522272b7406dfe; no voters:
I20260924 04:49:11.613763 13661 raft_consensus.cc:2781] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:49:11.613867 13661 raft_consensus.cc:482] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:49:11.613919 13661 raft_consensus.cc:3037] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:11.615301 13661 raft_consensus.cc:504] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94fbf3bb10a4e3688522272b7406dfe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "92d149a4bec34c0c9684812a9d290fa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33215 } } peers { permanent_uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42359 } }
I20260924 04:49:11.615694 13661 leader_election.cc:288] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [CANDIDATE]: Term 1 election: Requested vote from peers 92d149a4bec34c0c9684812a9d290fa5 (127.0.98.3:33215), 7033fc0ada4a4dcfa9038ef7dcb2c3b9 (127.0.98.2:42359)
I20260924 04:49:11.616148 13572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041729545d3d4da9a02294e2c2f780a5" candidate_uuid: "a94fbf3bb10a4e3688522272b7406dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d149a4bec34c0c9684812a9d290fa5"
I20260924 04:49:11.616284 13572 raft_consensus.cc:3037] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:11.617630 13572 raft_consensus.cc:2454] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a94fbf3bb10a4e3688522272b7406dfe in term 1.
I20260924 04:49:11.617892 13438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041729545d3d4da9a02294e2c2f780a5" candidate_uuid: "a94fbf3bb10a4e3688522272b7406dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9"
I20260924 04:49:11.617941 13237 leader_election.cc:302] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92d149a4bec34c0c9684812a9d290fa5, a94fbf3bb10a4e3688522272b7406dfe; no voters:
I20260924 04:49:11.618021 13438 raft_consensus.cc:3037] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:11.618108 13661 raft_consensus.cc:2781] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:49:11.618327 13661 raft_consensus.cc:686] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [term 1 LEADER]: Becoming Leader. State: Replica: a94fbf3bb10a4e3688522272b7406dfe, State: Running, Role: LEADER
I20260924 04:49:11.618691 13661 consensus_queue.cc:237] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94fbf3bb10a4e3688522272b7406dfe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "92d149a4bec34c0c9684812a9d290fa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33215 } } peers { permanent_uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42359 } }
I20260924 04:49:11.619357 13438 raft_consensus.cc:2454] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a94fbf3bb10a4e3688522272b7406dfe in term 1.
I20260924 04:49:11.622718 13151 catalog_manager.cc:5495] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe reported cstate change: term changed from 0 to 1, leader changed from <none> to a94fbf3bb10a4e3688522272b7406dfe (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a94fbf3bb10a4e3688522272b7406dfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94fbf3bb10a4e3688522272b7406dfe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92d149a4bec34c0c9684812a9d290fa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42359 } health_report { overall_health: UNKNOWN } } }
I20260924 04:49:11.685900 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:11.682375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.0.98.1:41071)
0924 04:49:11.682523 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:11.682527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:11.682541 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:49:11.682616 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:11.682618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:11.682671 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:11.682759 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:49:11.683047 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.683590 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.684421 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:11.684635 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:11.684721 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:11.685011 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:49:11.685446 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:49:11.685560 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:11.685682 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 04:49:11.685736 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:49:11.692430 13438 raft_consensus.cc:1261] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9 [term 1 FOLLOWER]: Refusing update from remote peer a94fbf3bb10a4e3688522272b7406dfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:49:11.692525 13572 raft_consensus.cc:1261] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5 [term 1 FOLLOWER]: Refusing update from remote peer a94fbf3bb10a4e3688522272b7406dfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:49:11.693130 13661 consensus_queue.cc:1035] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7033fc0ada4a4dcfa9038ef7dcb2c3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:49:11.693384 13671 consensus_queue.cc:1035] T 041729545d3d4da9a02294e2c2f780a5 P a94fbf3bb10a4e3688522272b7406dfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "92d149a4bec34c0c9684812a9d290fa5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:49:11.703320 13675 mvcc.cc:205] Tried to move back new op lower bound from 7332763040524853248 to 7332763040234418176. Current Snapshot: MvccSnapshot[applied={T|T < 7332763040524853248}]
I20260924 04:49:11.707808 13674 mvcc.cc:205] Tried to move back new op lower bound from 7332763040524853248 to 7332763040234418176. Current Snapshot: MvccSnapshot[applied={T|T < 7332763040524853248 or (T in {7332763040524853248})}]
I20260924 04:49:11.710457 13676 mvcc.cc:205] Tried to move back new op lower bound from 7332763040524853248 to 7332763040234418176. Current Snapshot: MvccSnapshot[applied={T|T < 7332763040524853248 or (T in {7332763040524853248})}]
I20260924 04:49:11.711289 13151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:49:11.711440 13151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:49:11.713613 13151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a8a06076fa8454295a66d83a34ec5a7: Sending DeleteTablet for 3 replicas of tablet 041729545d3d4da9a02294e2c2f780a5
I20260924 04:49:11.714320 13284 tablet_service.cc:1514] Processing DeleteTablet for tablet 041729545d3d4da9a02294e2c2f780a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51100
I20260924 04:49:11.714471 13418 tablet_service.cc:1514] Processing DeleteTablet for tablet 041729545d3d4da9a02294e2c2f780a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36178
I20260924 04:49:11.714447 13552 tablet_service.cc:1514] Processing DeleteTablet for tablet 041729545d3d4da9a02294e2c2f780a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60622
I20260924 04:49:11.715138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 13190
I20260924 04:49:11.716094 13684 tablet_replica.cc:330] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9: stopping tablet replica
I20260924 04:49:11.716501 13684 raft_consensus.cc:2229] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:49:11.716849 13684 raft_consensus.cc:2258] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:49:11.718412 13684 ts_tablet_manager.cc:1905] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:49:11.721369 13684 ts_tablet_manager.cc:1918] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:49:11.721560 13684 log.cc:1196] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/041729545d3d4da9a02294e2c2f780a5
I20260924 04:49:11.721959 13684 ts_tablet_manager.cc:1939] T 041729545d3d4da9a02294e2c2f780a5 P 7033fc0ada4a4dcfa9038ef7dcb2c3b9: Deleting consensus metadata
I20260924 04:49:11.722746 13685 tablet_replica.cc:330] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5: stopping tablet replica
I20260924 04:49:11.723196 13685 raft_consensus.cc:2229] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:49:11.723541 13136 catalog_manager.cc:4841] TS 7033fc0ada4a4dcfa9038ef7dcb2c3b9 (127.0.98.2:42359): tablet 041729545d3d4da9a02294e2c2f780a5 (table test-table [id=e2ca7e77231e4111a9e52dc7cbf7fcad]) successfully deleted
I20260924 04:49:11.723646 13685 raft_consensus.cc:2258] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 04:49:11.723909 13138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41071 (error 108)
I20260924 04:49:11.724162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 13354
I20260924 04:49:11.725630 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:11.724811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41071 (local address 127.0.0.1:51122)
0924 04:49:11.725469 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:11.725543 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569}
W20260924 04:49:11.725901 13138 catalog_manager.cc:4584] TS a94fbf3bb10a4e3688522272b7406dfe (127.0.98.1:41071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 041729545d3d4da9a02294e2c2f780a5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41071: connect: Connection refused (error 111)
I20260924 04:49:11.727826 13685 ts_tablet_manager.cc:1905] T 041729545d3d4da9a02294e2c2f780a5 P 92d149a4bec34c0c9684812a9d290fa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:49:11.729465 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 13487
I20260924 04:49:11.735651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 13119
I20260924 04:49:11.735689 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:11.735441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33215 (local address 127.0.0.1:60638)
0924 04:49:11.735560 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:11.735612 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:33215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 04:49:11.741189 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 04:49:12.076340 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 04:49:12.076467 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 04:49:12.411495 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 04:49:12.411 UTC [12186] LOG: received smart shutdown request
2026-09-24 04:49:12.415 UTC [12186] LOG: background worker "logical replication launcher" (PID 12195) exited with exit code 1
2026-09-24 04:49:12.416 UTC [12190] LOG: shutting down
2026-09-24 04:49:12.589 UTC [12186] LOG: database system is shut down
2026-09-24T04:49:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47999 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:49:12 dist-test-slave-hvcl krb5kdc[13695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:49:12 dist-test-slave-hvcl krb5kdc[13695](info): set up 2 sockets
Sep 24 04:49:12 dist-test-slave-hvcl krb5kdc[13695](info): commencing operation
krb5kdc: starting...
W20260924 04:49:13.658820 392 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:49:13 dist-test-slave-hvcl krb5kdc[13695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:49:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:49:13.781796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41029
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41029,127.0.98.61:43111,127.0.98.60:41959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:13.892740 13708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:13.893016 13708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:13.893074 13708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:13.897225 13708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:49:13.897291 13708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:13.897324 13708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:13.897348 13708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:49:13.897377 13708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:49:13.902752 13708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41029,127.0.98.61:43111,127.0.98.60:41959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:13.903879 13708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:13.904901 13708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:13.913509 13713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:13.913561 13708 server_base.cc:1034] running on GCE node
W20260924 04:49:13.913517 13716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:13.913561 13714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:13.914376 13708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:13.915552 13708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:13.916739 13708 hybrid_clock.cc:648] HybridClock initialized: now 1790225353916704 us; error 49 us; skew 500 ppm
Sep 24 04:49:13 dist-test-slave-hvcl krb5kdc[13695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:13.919584 13708 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:49:13.920544 13708 webserver.cc:460] Webserver started at http://127.0.98.62:39657/ using document root <none> and password file <none>
I20260924 04:49:13.921100 13708 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:13.921165 13708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:13.921363 13708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:49:13.923493 13708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "c6eea598b66845229df3846a03a64011"
format_stamp: "Formatted at 2026-09-24 04:49:13 on dist-test-slave-hvcl"
server_key: "b985a05aa91583b2d4424935369da226"
server_key_iv: "7d42d1b7a55575a1c20d22d620771609"
server_key_version: "encryptionkey@0"
I20260924 04:49:13.924049 13708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "c6eea598b66845229df3846a03a64011"
format_stamp: "Formatted at 2026-09-24 04:49:13 on dist-test-slave-hvcl"
server_key: "b985a05aa91583b2d4424935369da226"
server_key_iv: "7d42d1b7a55575a1c20d22d620771609"
server_key_version: "encryptionkey@0"
I20260924 04:49:13.927806 13708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:49:13.930235 13723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:13.931602 13708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:49:13.931717 13708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "c6eea598b66845229df3846a03a64011"
format_stamp: "Formatted at 2026-09-24 04:49:13 on dist-test-slave-hvcl"
server_key: "b985a05aa91583b2d4424935369da226"
server_key_iv: "7d42d1b7a55575a1c20d22d620771609"
server_key_version: "encryptionkey@0"
I20260924 04:49:13.931813 13708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:13.950621 13708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:13.954097 13708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:13.954298 13708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:13.961406 13708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41029
I20260924 04:49:13.961509 13775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41029 every 8 connection(s)
I20260924 04:49:13.962400 13708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:49:13.965236 13776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:13.968060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 13708
I20260924 04:49:13.968190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:49:13.968479 392 external_mini_cluster.cc:1428] Setting key 93af8a70833fa998fe68631f1cb7880c
I20260924 04:49:13.969491 13776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 04:49:13 dist-test-slave-hvcl krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:13.986536 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:13.971934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41029 (local address 127.0.0.1:54474)
0924 04:49:13.972479 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:13.972533 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:13.973470 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:13.974311 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:13.974316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:13.974692 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:13.974949 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:13.974965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:13.976631 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:13.976634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:13.977491 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:13.977501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:13.978172 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:13.978175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:13.978289 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:13.978938 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:13.978964 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:13.980701 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:13.983044 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:13.983053 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:13.983065 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:13.983422 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:13.983805 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:13.983808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:13.983809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:13.983936 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:13.984735 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:13.984741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:13.984882 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 04:49:13.985065 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 04:49:13.985481 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":99,"thread_start_us":143,"threads_started":1}
I20260924 04:49:13.986541 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:13.972887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.0.98.62:41029)
0924 04:49:13.973242 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:13.973254 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:13.973305 (+ 51us) server_negotiation.cc:407] Connection header received
0924 04:49:13.973809 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:13.973815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:13.974083 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:13.974298 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:13.975133 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:13.976511 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:13.977607 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:13.978004 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:13.978126 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:13.980871 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:13.980879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:13.980895 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:13.980923 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:13.982890 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:13.983536 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:13.983543 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:13.983545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:13.983693 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:13.984031 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:13.984034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:13.984036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:13.984610 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:13.984759 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:13.985166 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 04:49:13.985422 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":234,"thread_start_us":107,"threads_started":1}
I20260924 04:49:13.989037 13776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:49:13.989804 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:13.989548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43111 (local address 127.0.0.1:34894)
0924 04:49:13.989664 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:13.989735 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:49:13.990000 13724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43111: connect: Connection refused (error 111)
I20260924 04:49:13.990815 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:13.990620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43111 (local address 127.0.0.1:34906)
0924 04:49:13.990716 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:13.990761 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 04:49:13.990988 13776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43111: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43111: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:49:13.998126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43111
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41029,127.0.98.61:43111,127.0.98.60:41959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:49:14.040244 13776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } attempt: 1
I20260924 04:49:14.041152 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.040813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43111 (local address 127.0.0.1:34918)
0924 04:49:14.040983 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.041058 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 04:49:14.041882 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.041665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43111 (local address 127.0.0.1:34926)
0924 04:49:14.041761 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.041813 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 04:49:14.042109 13776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43111: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43111: connect: Connection refused (error 111)
W20260924 04:49:14.108518 13783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:14.108798 13783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:14.108837 13783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:49:14.110275 13776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } attempt: 2
I20260924 04:49:14.111099 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.110756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43111 (local address 127.0.0.1:34928)
0924 04:49:14.110910 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.111007 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 04:49:14.111796 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.111590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43111 (local address 127.0.0.1:34940)
0924 04:49:14.111683 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.111733 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 04:49:14.111902 13776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43111: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43111: connect: Connection refused (error 111)
W20260924 04:49:14.113371 13783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:49:14.113423 13783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:14.113442 13783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:14.113464 13783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:49:14.113484 13783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:49:14.118948 13783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41029,127.0.98.61:43111,127.0.98.60:41959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:14.120069 13783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:14.121099 13783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:14.129827 13789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:14.129844 13788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:14.129899 13783 server_base.cc:1034] running on GCE node
W20260924 04:49:14.129830 13791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:14.130666 13783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:14.131963 13783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:14.133121 13783 hybrid_clock.cc:648] HybridClock initialized: now 1790225354133094 us; error 46 us; skew 500 ppm
Sep 24 04:49:14 dist-test-slave-hvcl krb5kdc[13695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:14.136062 13783 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:49:14.137001 13783 webserver.cc:460] Webserver started at http://127.0.98.61:33603/ using document root <none> and password file <none>
I20260924 04:49:14.137511 13783 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:14.137566 13783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:14.137730 13783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:49:14.139792 13783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/data/instance:
uuid: "f9839818ea594f63841aa996707d5b18"
format_stamp: "Formatted at 2026-09-24 04:49:14 on dist-test-slave-hvcl"
server_key: "5e9b43b8673ca88d477d9620ce495f0d"
server_key_iv: "8eb3c3a1975d4f887ebca7a70229995e"
server_key_version: "encryptionkey@0"
I20260924 04:49:14.140291 13783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/wal/instance:
uuid: "f9839818ea594f63841aa996707d5b18"
format_stamp: "Formatted at 2026-09-24 04:49:14 on dist-test-slave-hvcl"
server_key: "5e9b43b8673ca88d477d9620ce495f0d"
server_key_iv: "8eb3c3a1975d4f887ebca7a70229995e"
server_key_version: "encryptionkey@0"
I20260924 04:49:14.143909 13783 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:49:14.146380 13798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:14.147761 13783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:49:14.147874 13783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/wal
uuid: "f9839818ea594f63841aa996707d5b18"
format_stamp: "Formatted at 2026-09-24 04:49:14 on dist-test-slave-hvcl"
server_key: "5e9b43b8673ca88d477d9620ce495f0d"
server_key_iv: "8eb3c3a1975d4f887ebca7a70229995e"
server_key_version: "encryptionkey@0"
I20260924 04:49:14.147970 13783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:14.170009 13783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:14.173353 13783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:14.173518 13783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:14.180423 13783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43111
I20260924 04:49:14.180460 13850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43111 every 8 connection(s)
I20260924 04:49:14.181396 13783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:49:14.184252 13851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:14.184486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 13783
I20260924 04:49:14.184600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-1/wal/instance
I20260924 04:49:14.184855 392 external_mini_cluster.cc:1428] Setting key 74b169924d1682a76d57bc0ae4637527
I20260924 04:49:14.188647 13851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 04:49:14 dist-test-slave-hvcl krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:14.203008 13776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } attempt: 3
I20260924 04:49:14.204684 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.191063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.0.98.62:41029)
0924 04:49:14.191222 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:14.191227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:14.192206 (+ 979us) server_negotiation.cc:407] Connection header received
0924 04:49:14.193208 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:14.193211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:14.193265 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:14.193346 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:14.194657 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.195499 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.196648 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.196892 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.196984 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.199905 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:14.199908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:14.199910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:14.199941 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:14.201893 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.202522 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.202525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.202527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.202613 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.203012 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.203016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.203018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.203241 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:14.203320 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:14.204281 (+ 961us) server_negotiation.cc:299] Negotiation successful
0924 04:49:14.204481 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:49:14.205546 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.191513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41029 (local address 127.0.0.1:54488)
0924 04:49:14.192026 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.192078 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:14.192990 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:14.193552 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:14.193561 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:14.193913 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:14.194462 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.194507 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.195633 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.195637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.196524 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.196534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.197038 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.197042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.197213 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.197803 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:14.197820 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:14.199710 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:14.202026 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.202032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.202053 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.202400 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.202708 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.202711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.202713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.202879 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.203343 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:14.203350 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:14.203998 (+ 648us) client_negotiation.cc:769] Sending connection context
0924 04:49:14.204207 (+ 209us) client_negotiation.cc:240] Negotiation successful
0924 04:49:14.204535 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":139,"threads_started":1}
I20260924 04:49:14.207023 13851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:49:14 dist-test-slave-hvcl krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Sep 24 04:49:14 dist-test-slave-hvcl krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:49:14.212325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41959
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41029,127.0.98.61:43111,127.0.98.60:41959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:49:14.213032 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.203447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43111 (local address 127.0.0.1:34948)
0924 04:49:14.203559 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.203573 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:14.203640 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:14.204316 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:14.204319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:14.204491 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:14.204615 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.204623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.206018 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.206021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.206680 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.206690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.207348 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.207352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.207443 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.207928 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:14.207942 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:14.209148 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:14.211200 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.211204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.211206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.211498 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.211809 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.211813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.211815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.211949 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.212495 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:14.212498 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:14.212602 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 04:49:14.212685 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 04:49:14.212828 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 04:49:14.213858 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.203568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.0.98.61:43111)
0924 04:49:14.203896 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:14.203907 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:14.203932 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:49:14.203972 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:14.203978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:14.204193 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:14.204317 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:14.204711 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.205910 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.206782 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.207242 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.207334 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.209259 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:14.209270 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:14.209286 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:14.209311 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:14.211102 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.211612 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.211619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.211621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.211707 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.212042 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.212046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.212048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.212370 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:14.213375 (+ 1005us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:14.213477 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 04:49:14.213650 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":117,"threads_started":1}
I20260924 04:49:14.215898 13776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:49:14.216042 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.207507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43111 (local address 127.0.0.1:34960)
0924 04:49:14.207629 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.207642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:14.207707 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:14.208014 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:14.208017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:14.208167 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:14.208243 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.208248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.209177 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.209180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.209973 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.209984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.210566 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.210570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.210705 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.211102 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:14.211116 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:14.212422 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:14.214814 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.214817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.214819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.215027 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.215245 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.215249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.215251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.215321 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.215628 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:14.215631 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:14.215711 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 04:49:14.215797 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:49:14.215917 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"spinlock_wait_cycles":768}
I20260924 04:49:14.216461 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.207558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.0.98.61:43111)
0924 04:49:14.207847 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:14.207853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:14.207863 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:49:14.207885 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:14.207886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:14.207917 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:14.207993 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:14.208335 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.208963 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.210089 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.210411 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.210538 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.212562 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:14.212565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:14.212568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:14.212586 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:14.214712 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.215098 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.215101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.215103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.215173 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.215390 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.215393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.215394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.215551 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:14.215629 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:14.216215 (+ 586us) server_negotiation.cc:299] Negotiation successful
0924 04:49:14.216316 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":122,"threads_started":1}
I20260924 04:49:14.216693 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.216426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47202)
0924 04:49:14.216563 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.216620 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 04:49:14.217047 13851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:49:14.217545 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.217306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47216)
0924 04:49:14.217446 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.217490 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 04:49:14.217737 13776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41959: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
I20260924 04:49:14.217998 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.217567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47224)
0924 04:49:14.217840 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.217932 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
W20260924 04:49:14.218209 13802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
I20260924 04:49:14.219105 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.218861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47226)
0924 04:49:14.218965 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.219042 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 04:49:14.219257 13851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41959: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
I20260924 04:49:14.248903 13776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } attempt: 1
I20260924 04:49:14.249759 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.249458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47234)
0924 04:49:14.249611 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.249675 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 04:49:14.250520 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.250301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47250)
0924 04:49:14.250414 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.250463 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 04:49:14.250643 13776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41959: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
I20260924 04:49:14.268528 13851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } attempt: 1
I20260924 04:49:14.269454 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.269086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47254)
0924 04:49:14.269268 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.269355 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 04:49:14.270182 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.269965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47268)
0924 04:49:14.270062 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.270113 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 04:49:14.270346 13851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41959: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
W20260924 04:49:14.324973 13858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:14.325268 13858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:14.325327 13858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:49:14.325805 13776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } attempt: 2
I20260924 04:49:14.326627 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.326335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47278)
0924 04:49:14.326487 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.326549 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 04:49:14.327329 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.327120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47294)
0924 04:49:14.327230 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.327274 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 04:49:14.327440 13776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41959: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
W20260924 04:49:14.329806 13858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:49:14.329862 13858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:14.329888 13858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:14.329919 13858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:49:14.329947 13858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:49:14.335561 13858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41029,127.0.98.61:43111,127.0.98.60:41959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:14.336675 13858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:14.337744 13858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:49:14.338510 13851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } attempt: 2
I20260924 04:49:14.339298 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.338957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47298)
0924 04:49:14.339133 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.339206 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 04:49:14.340057 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.339832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47314)
0924 04:49:14.339939 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.339993 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:49:14.340196 13851 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41959: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41959: connect: Connection refused (error 111)
W20260924 04:49:14.346314 13865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:14.346287 13867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:14.346335 13864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:14.346630 13858 server_base.cc:1034] running on GCE node
I20260924 04:49:14.347349 13858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:14.348547 13858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:14.349712 13858 hybrid_clock.cc:648] HybridClock initialized: now 1790225354349687 us; error 38 us; skew 500 ppm
Sep 24 04:49:14 dist-test-slave-hvcl krb5kdc[13695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:14.352730 13858 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:49:14.353699 13858 webserver.cc:460] Webserver started at http://127.0.98.60:37755/ using document root <none> and password file <none>
I20260924 04:49:14.354243 13858 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:14.354310 13858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:14.354521 13858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:49:14.356673 13858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/data/instance:
uuid: "cef4a10a392d47dd9ff8f3a0f6317e51"
format_stamp: "Formatted at 2026-09-24 04:49:14 on dist-test-slave-hvcl"
server_key: "b136865104bee6e087a95fbd0affc1d1"
server_key_iv: "9a8b2ccb26b56c875adac383b1fb79a2"
server_key_version: "encryptionkey@0"
I20260924 04:49:14.357250 13858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/wal/instance:
uuid: "cef4a10a392d47dd9ff8f3a0f6317e51"
format_stamp: "Formatted at 2026-09-24 04:49:14 on dist-test-slave-hvcl"
server_key: "b136865104bee6e087a95fbd0affc1d1"
server_key_iv: "9a8b2ccb26b56c875adac383b1fb79a2"
server_key_version: "encryptionkey@0"
I20260924 04:49:14.361253 13858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:49:14.363932 13874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:14.365417 13858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:49:14.365557 13858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/wal
uuid: "cef4a10a392d47dd9ff8f3a0f6317e51"
format_stamp: "Formatted at 2026-09-24 04:49:14 on dist-test-slave-hvcl"
server_key: "b136865104bee6e087a95fbd0affc1d1"
server_key_iv: "9a8b2ccb26b56c875adac383b1fb79a2"
server_key_version: "encryptionkey@0"
I20260924 04:49:14.365675 13858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:14.402866 13858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:14.406148 13858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:14.406347 13858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:14.413419 13858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41959
I20260924 04:49:14.413538 13926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41959 every 8 connection(s)
I20260924 04:49:14.414445 13858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:49:14.417371 13927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:14.419214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 13858
I20260924 04:49:14.419351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225194677594-392-0/minicluster-data/master-2/wal/instance
I20260924 04:49:14.419616 392 external_mini_cluster.cc:1428] Setting key 9b1cac7b2e94cccaad83759720d5ebfb
I20260924 04:49:14.421679 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:49:14.426594 13776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } attempt: 3
Sep 24 04:49:14 dist-test-slave-hvcl krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:49:14 dist-test-slave-hvcl krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:14.431452 13851 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } attempt: 3
Sep 24 04:49:14 dist-test-slave-hvcl krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:49:14.436177 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.424281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.0.98.62:41029)
0924 04:49:14.424424 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:14.424429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:14.424814 (+ 385us) server_negotiation.cc:407] Connection header received
0924 04:49:14.424932 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:14.424936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:14.424983 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:14.425056 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:14.426681 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.427307 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.428496 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.428701 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.428779 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.432212 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:14.432218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:14.432220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:14.432246 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:14.433993 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.434605 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.434609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.434611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.434723 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.435056 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.435059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.435061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.435327 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:14.435394 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:14.435880 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 04:49:14.436009 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:49:14.436503 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.427061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47330)
0924 04:49:14.427505 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.427520 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:14.427587 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:14.427972 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:14.427975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:14.428161 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:14.428253 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.428259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.429312 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.429316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.429905 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.429914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.430449 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.430453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.430541 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.431032 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:14.431046 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:14.432472 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:14.434911 (+ 2439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.434914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.434917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.435191 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.435584 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.435588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.435589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.435674 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.436066 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:14.436069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:14.436159 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 04:49:14.436233 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:49:14.436364 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365}
I20260924 04:49:14.437269 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.427246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.0.98.60:41959)
0924 04:49:14.427763 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:14.427771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:14.427781 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:49:14.427809 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:14.427811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:14.427852 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:14.427949 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:14.428371 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.429223 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.430017 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.430328 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.430438 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.432623 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:14.432627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:14.432630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:14.432654 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:14.434786 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.435338 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.435341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.435342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.435480 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.435773 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.435776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.435777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.435961 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:14.436752 (+ 791us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:14.436826 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:49:14.436947 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":118,"threads_started":1}
I20260924 04:49:14.437269 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.421004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.0.98.60:41959)
0924 04:49:14.421465 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:14.421485 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:14.421611 (+ 126us) server_negotiation.cc:407] Connection header received
0924 04:49:14.422610 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:14.422640 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:14.423170 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:14.423517 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:14.424310 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.425759 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.426661 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.427406 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.427576 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.430244 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:14.430250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:14.430262 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:14.430285 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:14.432606 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.433407 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.433414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.433433 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.433586 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.433915 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.433919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.433920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.434757 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:14.434953 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:14.435806 (+ 853us) server_negotiation.cc:299] Negotiation successful
0924 04:49:14.436124 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":296,"thread_start_us":121,"threads_started":1}
I20260924 04:49:14.437252 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.424163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41029 (local address 127.0.0.1:54492)
0924 04:49:14.424678 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.424726 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:14.424845 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:14.425117 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:14.425122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:14.425550 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:14.425762 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.425769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.427423 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.427427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.428401 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.428410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.428831 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.428835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.428947 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.430105 (+ 1158us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:14.430124 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:14.432023 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:14.434108 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.434117 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.434123 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.434469 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.434826 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.434829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.434831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.434927 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.435473 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:14.435480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:14.435623 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 04:49:14.436356 (+ 733us) client_negotiation.cc:240] Negotiation successful
0924 04:49:14.436538 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":6,"thread_start_us":154,"threads_started":1}
Sep 24 04:49:14 dist-test-slave-hvcl krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:49:14.438700 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:49:14.441809 13776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011: Bootstrap starting.
I20260924 04:49:14.443094 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.431906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47334)
0924 04:49:14.432107 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.432122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:14.432202 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:14.433236 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:14.433240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:14.433411 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:14.433554 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.433561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.434807 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.434811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.435673 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.435684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.436551 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.436555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.436648 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.437083 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:14.437096 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:14.438943 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:14.441462 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.441465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.441467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.441711 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.442031 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.442034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.442036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.442132 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.442550 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:14.442553 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:14.442652 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 04:49:14.442757 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 04:49:14.442890 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260924 04:49:14.443130 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.432404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47334 (local address 127.0.98.60:41959)
0924 04:49:14.432985 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:14.432995 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:14.433008 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:49:14.433039 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:14.433042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:14.433082 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:14.433667 (+ 585us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:14.433779 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.434679 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.435848 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.436148 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.439137 (+ 2989us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.439240 (+ 103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:14.439243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:14.439245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:14.439263 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:14.441348 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.441816 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.441819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.441821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.441909 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.442225 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.442228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.442230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.442444 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:14.442537 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:14.442816 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 04:49:14.442936 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":364,"threads_started":1}
I20260924 04:49:14.445150 13776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011: Neither blocks nor log segments found. Creating new log.
Sep 24 04:49:14 dist-test-slave-hvcl krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:14.446056 13776 log.cc:824] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:14.446319 13851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18: Bootstrap starting.
I20260924 04:49:14.449064 13851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:14.449259 13776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011: No bootstrap required, opened a new log
I20260924 04:49:14.449504 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.439810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.0.98.62:41029)
0924 04:49:14.439926 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:14.439930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:14.439974 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:49:14.440108 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:14.440111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:14.440156 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:14.440230 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:14.441518 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.442213 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.443530 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.443841 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.443909 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.446169 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:14.446173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:14.446176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:14.446205 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:14.447577 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.448051 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.448055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.448056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.448140 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.448420 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.448424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.448426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.448633 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:14.448769 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:14.449130 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 04:49:14.449249 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:49:14.449966 13851 log.cc:824] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:49:14 dist-test-slave-hvcl krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:49:14.452160 13851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18: No bootstrap required, opened a new log
I20260924 04:49:14.453480 13776 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } }
I20260924 04:49:14.453713 13776 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:14.453769 13776 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6eea598b66845229df3846a03a64011, State: Initialized, Role: FOLLOWER
I20260924 04:49:14.454279 13776 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } }
I20260924 04:49:14.454908 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.439547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.0.98.61:43111)
0924 04:49:14.443125 (+ 3578us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:14.443129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:14.443143 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:49:14.443176 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:14.443178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:14.443220 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:14.443293 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:14.446991 (+ 3698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.447736 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.448786 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.449140 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.449219 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.451530 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:14.451534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:14.451537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:14.451564 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:14.453118 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.453594 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.453598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.453600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.453685 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.453994 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.453997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.453999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.454267 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:14.454368 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:14.454649 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 04:49:14.454772 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3491}
I20260924 04:49:14.454909 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.439182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43111 (local address 127.0.0.1:34966)
0924 04:49:14.439570 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.439584 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:14.439655 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:14.446525 (+ 6870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:14.446528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:14.446693 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:14.446817 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.446825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.447881 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.447884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.448652 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.448664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.449382 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.449386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.449487 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.449907 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:14.449921 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:14.451374 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:14.453240 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.453243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.453245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.453493 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.453791 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.453794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.453796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.453885 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.454371 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:14.454374 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:14.454479 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 04:49:14.454586 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 04:49:14.454732 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":111,"threads_started":1}
I20260924 04:49:14.455364 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } } }
I20260924 04:49:14.455504 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:49:14.455822 13776 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:49:14.455964 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } has no permanent_uuid. Determining permanent_uuid...
W20260924 04:49:14.456014 13729 tablet.cc:2369] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:49:14.456183 13851 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } }
I20260924 04:49:14.456375 13851 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:14.456439 13851 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9839818ea594f63841aa996707d5b18, State: Initialized, Role: FOLLOWER
I20260924 04:49:14.457650 13851 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } }
I20260924 04:49:14.458999 13851 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 04:49:14 dist-test-slave-hvcl krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:49:14.460027 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } } }
I20260924 04:49:14.462149 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 04:49:14 dist-test-slave-hvcl krb5kdc[13695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:49:14.460696 13708 master_runner.cc:186] Error getting master registration for 127.0.98.61:43111: OK, instance_id { permanent_uuid: "f9839818ea594f63841aa996707d5b18" instance_seqno: 1790225354178498 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:49:14.464620 13708 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:49:14.464989 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.456405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41959 (local address 127.0.0.1:47346)
0924 04:49:14.456625 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:14.456639 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:14.456723 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:14.457002 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:14.457005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:14.457129 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:14.457187 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.457192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.458156 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.458158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.458873 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:14.458884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.459554 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.459558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:14.459629 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.460110 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:14.460126 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:14.461631 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:14.463499 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.463501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.463503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.463722 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.463980 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:14.463982 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:14.463984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:14.464058 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:14.464473 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:14.464475 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:14.464587 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 04:49:14.464684 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 04:49:14.464818 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260924 04:49:14.465404 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.456628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.0.98.60:41959)
0924 04:49:14.456722 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:14.456726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:14.456752 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:49:14.456790 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:14.456792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:14.456832 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:14.456910 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:14.457296 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.458050 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.459025 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.459360 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.459455 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.461759 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:14.461762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:14.461764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:14.461786 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:14.463394 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.463812 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.463815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.463816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.463889 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.464153 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.464156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.464157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.464362 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:14.464443 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:14.465171 (+ 728us) server_negotiation.cc:299] Negotiation successful
0924 04:49:14.465279 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 04:49:14.468113 13948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:49:14.468189 13948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:49:14.468385 13927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51: Bootstrap starting.
I20260924 04:49:14.465855 13783 master_runner.cc:186] Error getting master registration for 127.0.98.60:41959: OK, instance_id { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" instance_seqno: 1790225354411354 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260924 04:49:14.469003 13960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:49:14.469058 13960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:49:14.469101 13783 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:49:14.471674 13927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:14.472707 13927 log.cc:824] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:14.475431 13927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51: No bootstrap required, opened a new log
I20260924 04:49:14.480449 13927 raft_consensus.cc:348] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } }
I20260924 04:49:14.480837 13927 raft_consensus.cc:374] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:14.480865 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.452030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34976 (local address 127.0.98.61:43111)
0924 04:49:14.455706 (+ 3676us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:14.455711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:14.455723 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:49:14.455755 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:14.455757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:14.455796 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:14.455868 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:14.457846 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.458514 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.459567 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.460160 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.460227 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.463752 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:14.463756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:14.463759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:14.463784 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:14.471356 (+ 7572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.479183 (+ 7827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.479189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.479191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.479336 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.479728 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.479732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.479734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.479987 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:14.480179 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:14.480547 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 04:49:14.480695 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3615}
I20260924 04:49:14.481118 13927 raft_consensus.cc:729] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef4a10a392d47dd9ff8f3a0f6317e51, State: Initialized, Role: FOLLOWER
I20260924 04:49:14.482762 13927 consensus_queue.cc:260] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } }
I20260924 04:49:14.483647 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } } }
I20260924 04:49:14.483803 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:49:14.484668 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:49:14.486877 13927 sys_catalog.cc:564] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:49:14.492652 13981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:49:14.492769 13981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:49:14.498046 13858 master_runner.cc:263] No leader master found from master cef4a10a392d47dd9ff8f3a0f6317e51
I20260924 04:49:14.498135 13858 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:49:14.509788 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.486879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.0.98.62:41029)
0924 04:49:14.487051 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:14.487056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:14.488179 (+ 1123us) server_negotiation.cc:407] Connection header received
0924 04:49:14.488354 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:14.488357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:14.488406 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:14.488456 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:14.489757 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.490443 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.496802 (+ 6359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.497166 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.497267 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.501349 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:14.501353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:14.501356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:14.501387 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:14.503539 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.505791 (+ 2252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.505794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.505796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.505885 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.506173 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.506177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.506178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.506402 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:14.506516 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:14.509480 (+ 2964us) server_negotiation.cc:299] Negotiation successful
0924 04:49:14.509614 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 04:49:14.510526 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.487608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.0.98.61:43111)
0924 04:49:14.487714 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:14.487718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:14.489308 (+ 1590us) server_negotiation.cc:407] Connection header received
0924 04:49:14.489420 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:14.489423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:14.489471 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:14.489544 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:14.494834 (+ 5290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.495648 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.499270 (+ 3622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.499624 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.499717 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.502363 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:14.502367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:14.502369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:14.502399 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:14.505330 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.507205 (+ 1875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.507209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.507211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.507303 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.508550 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.508553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.508556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.508772 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:14.508873 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:14.510252 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0924 04:49:14.510375 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:49:14.521322 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:14.494539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47362 (local address 127.0.98.60:41959)
0924 04:49:14.494704 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:14.494709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:14.507144 (+ 12435us) server_negotiation.cc:407] Connection header received
0924 04:49:14.507714 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:14.507717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:14.507769 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:14.507817 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:14.510897 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.511752 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.513015 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:14.513367 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:14.513446 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:14.517610 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:14.517614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:14.517617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:14.517651 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:14.519380 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.519885 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.519889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.519891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.519989 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:14.520275 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:14.520278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:14.520280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:14.520517 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:14.520628 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:14.521001 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 04:49:14.521134 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 04:49:14.602314 13937 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:49:14.602559 13937 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } }
I20260924 04:49:14.603880 13937 leader_election.cc:288] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9839818ea594f63841aa996707d5b18 (127.0.98.61:43111), cef4a10a392d47dd9ff8f3a0f6317e51 (127.0.98.60:41959)
I20260924 04:49:14.604815 13901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6eea598b66845229df3846a03a64011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" is_pre_election: true
I20260924 04:49:14.605199 13901 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6eea598b66845229df3846a03a64011 in term 0.
I20260924 04:49:14.606024 13825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6eea598b66845229df3846a03a64011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9839818ea594f63841aa996707d5b18" is_pre_election: true
I20260924 04:49:14.606369 13825 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6eea598b66845229df3846a03a64011 in term 0.
I20260924 04:49:14.606958 13727 leader_election.cc:302] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6eea598b66845229df3846a03a64011, cef4a10a392d47dd9ff8f3a0f6317e51; no voters:
I20260924 04:49:14.607862 13937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:49:14.607963 13937 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:49:14.608021 13937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:14.609407 13937 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } }
I20260924 04:49:14.610210 13825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6eea598b66845229df3846a03a64011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9839818ea594f63841aa996707d5b18"
I20260924 04:49:14.610404 13825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:14.611848 13825 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6eea598b66845229df3846a03a64011 in term 1.
I20260924 04:49:14.612247 13724 leader_election.cc:302] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6eea598b66845229df3846a03a64011, f9839818ea594f63841aa996707d5b18; no voters:
I20260924 04:49:14.613042 13937 leader_election.cc:288] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [CANDIDATE]: Term 1 election: Requested vote from peers f9839818ea594f63841aa996707d5b18 (127.0.98.61:43111), cef4a10a392d47dd9ff8f3a0f6317e51 (127.0.98.60:41959)
I20260924 04:49:14.613157 13937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:49:14.613231 13937 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [term 1 LEADER]: Becoming Leader. State: Replica: c6eea598b66845229df3846a03a64011, State: Running, Role: LEADER
I20260924 04:49:14.613581 13937 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } }
I20260924 04:49:14.614470 13901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6eea598b66845229df3846a03a64011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51"
I20260924 04:49:14.614609 13901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:14.615830 13901 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6eea598b66845229df3846a03a64011 in term 1.
I20260924 04:49:14.617651 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6eea598b66845229df3846a03a64011. Latest consensus state: current_term: 1 leader_uuid: "c6eea598b66845229df3846a03a64011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } } }
I20260924 04:49:14.617786 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:14.620455 13986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:49:14.621444 13986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:49:14.628708 13901 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [term 1 FOLLOWER]: Refusing update from remote peer c6eea598b66845229df3846a03a64011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:49:14.629467 13984 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:49:14.637269 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6eea598b66845229df3846a03a64011. Latest consensus state: current_term: 1 leader_uuid: "c6eea598b66845229df3846a03a64011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } } }
I20260924 04:49:14.637434 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:49:14.639348 13825 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [term 1 FOLLOWER]: Refusing update from remote peer c6eea598b66845229df3846a03a64011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:49:14.639974 13984 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:49:14.644555 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6eea598b66845229df3846a03a64011. Latest consensus state: current_term: 1 leader_uuid: "c6eea598b66845229df3846a03a64011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } } }
I20260924 04:49:14.644680 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:49:14.648536 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6eea598b66845229df3846a03a64011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } } }
I20260924 04:49:14.648679 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:14.649259 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6eea598b66845229df3846a03a64011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } } }
I20260924 04:49:14.649379 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:49:14.649744 13987 mvcc.cc:205] Tried to move back new op lower bound from 7332763052550586368 to 7332763052509634560. Current Snapshot: MvccSnapshot[applied={T|T < 7332763052550586368}]
I20260924 04:49:14.649888 13990 mvcc.cc:205] Tried to move back new op lower bound from 7332763052550586368 to 7332763052509634560. Current Snapshot: MvccSnapshot[applied={T|T < 7332763052550586368}]
I20260924 04:49:14.652261 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6eea598b66845229df3846a03a64011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } } }
I20260924 04:49:14.652411 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:14.654304 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6eea598b66845229df3846a03a64011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eea598b66845229df3846a03a64011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } peers { permanent_uuid: "f9839818ea594f63841aa996707d5b18" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43111 } } peers { permanent_uuid: "cef4a10a392d47dd9ff8f3a0f6317e51" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41959 } } }
I20260924 04:49:14.654421 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:49:14.656562 13986 catalog_manager.cc:1347] Generated new cluster ID: b9f85683f3a64f6ba95ee1a86c8324f9
I20260924 04:49:14.656628 13986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260924 04:49:14.657785 13880 tablet.cc:2369] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:49:14.664027 13988 mvcc.cc:205] Tried to move back new op lower bound from 7332763052550586368 to 7332763052509634560. Current Snapshot: MvccSnapshot[applied={T|T < 7332763052550586368}]
I20260924 04:49:14.666553 13986 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:49:14.667654 13986 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:49:14.681557 13986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6eea598b66845229df3846a03a64011: Generated new TSK 0
I20260924 04:49:14.682268 13986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:49:15.470302 13960 catalog_manager.cc:1259] Loaded cluster ID: b9f85683f3a64f6ba95ee1a86c8324f9
I20260924 04:49:15.470378 13960 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18: loading cluster ID for follower catalog manager: success
I20260924 04:49:15.472103 13960 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18: acquiring CA information for follower catalog manager: success
I20260924 04:49:15.473305 13960 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f9839818ea594f63841aa996707d5b18: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:49:15.494024 13981 catalog_manager.cc:1259] Loaded cluster ID: b9f85683f3a64f6ba95ee1a86c8324f9
I20260924 04:49:15.494109 13981 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51: loading cluster ID for follower catalog manager: success
I20260924 04:49:15.495795 13981 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51: acquiring CA information for follower catalog manager: success
I20260924 04:49:15.496920 13981 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cef4a10a392d47dd9ff8f3a0f6317e51: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:49:15.740568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 13708
I20260924 04:49:15.746047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 13783
I20260924 04:49:15.751039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 13858
2026-09-24T04:49:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3068 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:49:15 dist-test-slave-hvcl krb5kdc[14001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:49:15 dist-test-slave-hvcl krb5kdc[14001](info): set up 2 sockets
Sep 24 04:49:15 dist-test-slave-hvcl krb5kdc[14001](info): commencing operation
krb5kdc: starting...
W20260924 04:49:16.715051 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:49:16 dist-test-slave-hvcl krb5kdc[14001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:49:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:49:16.835565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:16.945809 14014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:16.946094 14014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:16.946136 14014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:16.950209 14014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:49:16.950279 14014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:16.950297 14014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:16.950316 14014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:49:16.950330 14014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:49:16.955798 14014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:16.956856 14014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:16.957842 14014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:16.966514 14020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:16.966583 14019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:16.966600 14014 server_base.cc:1034] running on GCE node
W20260924 04:49:16.966527 14022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:16.967424 14014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:16.968561 14014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:16.969741 14014 hybrid_clock.cc:648] HybridClock initialized: now 1790225356969721 us; error 35 us; skew 500 ppm
Sep 24 04:49:16 dist-test-slave-hvcl krb5kdc[14001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:16.972642 14014 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:49:16.973554 14014 webserver.cc:460] Webserver started at http://127.0.98.62:40465/ using document root <none> and password file <none>
I20260924 04:49:16.974107 14014 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:16.974166 14014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:16.974431 14014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:49:16.976606 14014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "6ff7dfce05be457c813c17ab8a8a3271"
format_stamp: "Formatted at 2026-09-24 04:49:16 on dist-test-slave-hvcl"
server_key: "9e470315de1ddd97f5b8a842853048d4"
server_key_iv: "fe27b3aecf03d9483b9bc22101886639"
server_key_version: "encryptionkey@0"
I20260924 04:49:16.977157 14014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ff7dfce05be457c813c17ab8a8a3271"
format_stamp: "Formatted at 2026-09-24 04:49:16 on dist-test-slave-hvcl"
server_key: "9e470315de1ddd97f5b8a842853048d4"
server_key_iv: "fe27b3aecf03d9483b9bc22101886639"
server_key_version: "encryptionkey@0"
I20260924 04:49:16.981158 14014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:49:16.983814 14029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:16.985261 14014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:49:16.985462 14014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "6ff7dfce05be457c813c17ab8a8a3271"
format_stamp: "Formatted at 2026-09-24 04:49:16 on dist-test-slave-hvcl"
server_key: "9e470315de1ddd97f5b8a842853048d4"
server_key_iv: "fe27b3aecf03d9483b9bc22101886639"
server_key_version: "encryptionkey@0"
I20260924 04:49:16.985575 14014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:17.018290 14014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:17.021689 14014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:17.021875 14014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:17.028746 14014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37957
I20260924 04:49:17.028787 14081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37957 every 8 connection(s)
I20260924 04:49:17.029721 14014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:49:17.032927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 14014
I20260924 04:49:17.033033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:49:17.033052 14082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:17.033293 392 external_mini_cluster.cc:1428] Setting key b46d293ff437f7bddf928268af1a62fe
I20260924 04:49:17.041154 14082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271: Bootstrap starting.
Sep 24 04:49:17 dist-test-slave-hvcl krb5kdc[14001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:17.044117 14082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:17.045001 14082 log.cc:824] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:17.047191 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:17.034500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49734 (local address 127.0.98.62:37957)
0924 04:49:17.034893 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:17.034908 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:17.034941 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:49:17.036046 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:17.036065 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:17.036402 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:17.036728 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:17.037549 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:17.038647 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:17.039568 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:17.040064 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:17.040211 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:17.042709 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:17.042715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:17.042730 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:17.042761 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:17.044380 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:17.044800 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:17.044805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:17.044811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:17.044918 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:17.045207 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:17.045210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:17.045212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:17.045622 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:17.045797 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:17.046091 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 04:49:17.046404 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
I20260924 04:49:17.047385 14082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271: No bootstrap required, opened a new log
I20260924 04:49:17.049856 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:49:17.052135 14082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff7dfce05be457c813c17ab8a8a3271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37957 } }
I20260924 04:49:17.052452 14082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:17.052510 14082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff7dfce05be457c813c17ab8a8a3271, State: Initialized, Role: FOLLOWER
I20260924 04:49:17.053001 14082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff7dfce05be457c813c17ab8a8a3271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37957 } }
I20260924 04:49:17.053164 14082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:49:17.053224 14082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:49:17.053321 14082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:17.054503 14082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff7dfce05be457c813c17ab8a8a3271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37957 } }
I20260924 04:49:17.054805 14082 leader_election.cc:302] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ff7dfce05be457c813c17ab8a8a3271; no voters:
I20260924 04:49:17.055117 14082 leader_election.cc:288] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:49:17.055235 14087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:49:17.055441 14087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [term 1 LEADER]: Becoming Leader. State: Replica: 6ff7dfce05be457c813c17ab8a8a3271, State: Running, Role: LEADER
I20260924 04:49:17.055779 14087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff7dfce05be457c813c17ab8a8a3271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37957 } }
I20260924 04:49:17.056267 14082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:49:17.057271 14089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ff7dfce05be457c813c17ab8a8a3271. Latest consensus state: current_term: 1 leader_uuid: "6ff7dfce05be457c813c17ab8a8a3271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff7dfce05be457c813c17ab8a8a3271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37957 } } }
I20260924 04:49:17.057412 14089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:17.057790 14096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:49:17.057753 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ff7dfce05be457c813c17ab8a8a3271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff7dfce05be457c813c17ab8a8a3271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37957 } } }
I20260924 04:49:17.058456 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:17.060551 14096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:49:17.065873 14096 catalog_manager.cc:1347] Generated new cluster ID: a264223572794d2595af9a0da1e7f4fd
I20260924 04:49:17.065961 14096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:49:17.081136 14096 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:49:17.082370 14096 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:49:17.088492 14096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ff7dfce05be457c813c17ab8a8a3271: Generated new TSK 0
I20260924 04:49:17.089395 14096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:49:17.105561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 14014
2026-09-24T04:49:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1355 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:49:17 dist-test-slave-hvcl krb5kdc[14109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:49:17 dist-test-slave-hvcl krb5kdc[14109](info): set up 2 sockets
Sep 24 04:49:17 dist-test-slave-hvcl krb5kdc[14109](info): commencing operation
krb5kdc: starting...
W20260924 04:49:18.066937 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:49:18 dist-test-slave-hvcl krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:49:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:49:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:49:18.189994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33651
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:18.299432 14122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:18.299710 14122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:18.299770 14122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:18.303819 14122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:49:18.303889 14122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:18.303915 14122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:18.303937 14122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:49:18.303966 14122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:49:18.309306 14122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:18.310427 14122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:18.311523 14122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:18.320493 14127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:18.320495 14128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:18.320573 14122 server_base.cc:1034] running on GCE node
W20260924 04:49:18.320541 14130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:18.321372 14122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:18.322544 14122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:18.323714 14122 hybrid_clock.cc:648] HybridClock initialized: now 1790225358323685 us; error 41 us; skew 500 ppm
Sep 24 04:49:18 dist-test-slave-hvcl krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:18.330058 14122 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:49:18.331161 14122 webserver.cc:460] Webserver started at http://127.0.98.62:45239/ using document root <none> and password file <none>
I20260924 04:49:18.331760 14122 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:18.331831 14122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:18.332038 14122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:49:18.334133 14122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "64d60a82d7e74b81b1e2739c1140b868"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "082b86a05e9dc0a37573877508db8cb4"
server_key_iv: "7c9b4ab5bcb7c54780f61f558c15a02c"
server_key_version: "encryptionkey@0"
I20260924 04:49:18.334702 14122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "64d60a82d7e74b81b1e2739c1140b868"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "082b86a05e9dc0a37573877508db8cb4"
server_key_iv: "7c9b4ab5bcb7c54780f61f558c15a02c"
server_key_version: "encryptionkey@0"
I20260924 04:49:18.338600 14122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:49:18.341449 14137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:18.343065 14122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:49:18.343214 14122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "64d60a82d7e74b81b1e2739c1140b868"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "082b86a05e9dc0a37573877508db8cb4"
server_key_iv: "7c9b4ab5bcb7c54780f61f558c15a02c"
server_key_version: "encryptionkey@0"
I20260924 04:49:18.343338 14122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:18.359800 14122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:18.363298 14122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:18.363497 14122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:18.370533 14122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33651
I20260924 04:49:18.370842 14189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33651 every 8 connection(s)
I20260924 04:49:18.371534 14122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:49:18.374346 14190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:18.376468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 14122
I20260924 04:49:18.376595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:49:18.376909 392 external_mini_cluster.cc:1428] Setting key 2201ac8a74b7ea895f59ad5f22f1a69e
I20260924 04:49:18.380599 14190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Bootstrap starting.
I20260924 04:49:18.382812 14190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:18.383580 14190 log.cc:824] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:18.385555 14190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: No bootstrap required, opened a new log
Sep 24 04:49:18 dist-test-slave-hvcl krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:18.388566 14190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } }
I20260924 04:49:18.388784 14190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:18.388818 14190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d60a82d7e74b81b1e2739c1140b868, State: Initialized, Role: FOLLOWER
I20260924 04:49:18.389251 14190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } }
I20260924 04:49:18.389384 14190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:49:18.389442 14190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:49:18.389534 14190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:18.390458 14190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } }
I20260924 04:49:18.390794 14190 leader_election.cc:302] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64d60a82d7e74b81b1e2739c1140b868; no voters:
I20260924 04:49:18.391101 14190 leader_election.cc:288] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:49:18.391230 14195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:49:18.391383 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:18.378360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.0.98.62:33651)
0924 04:49:18.378869 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:18.378885 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:18.378922 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:49:18.379696 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:18.379716 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:18.380081 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:18.380411 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:18.381216 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:18.382213 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:18.383122 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:18.383595 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:18.383743 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:18.386216 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:18.386222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:18.386235 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:18.386263 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:18.388323 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:18.388739 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:18.388744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:18.388749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:18.388906 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:18.389168 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:18.389171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:18.389173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:18.389649 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:18.389857 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:18.390134 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 04:49:18.390422 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":134,"threads_started":1}
I20260924 04:49:18.391441 14195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 1 LEADER]: Becoming Leader. State: Replica: 64d60a82d7e74b81b1e2739c1140b868, State: Running, Role: LEADER
I20260924 04:49:18.391929 14195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } }
I20260924 04:49:18.392397 14190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:49:18.393580 14197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64d60a82d7e74b81b1e2739c1140b868. Latest consensus state: current_term: 1 leader_uuid: "64d60a82d7e74b81b1e2739c1140b868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } } }
I20260924 04:49:18.393937 14197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:18.393843 14196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64d60a82d7e74b81b1e2739c1140b868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } } }
I20260924 04:49:18.394266 14196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: This master's current role is: LEADER
W20260924 04:49:18.397258 14210 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:49:18.397325 14210 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:49:18.397424 14209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:49:18.398077 14209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:49:18.404157 14209 catalog_manager.cc:1347] Generated new cluster ID: 6e2db1f69d4d43a48e50e048afe00f74
I20260924 04:49:18.404353 14209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:49:18.427731 14209 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:49:18.428997 14209 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:49:18.436049 14209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Generated new TSK 0
I20260924 04:49:18.436906 14209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:49:18.481765 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33651
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:18.617337 14216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:18.617599 14216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:18.617641 14216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:18.621841 14216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:18.621938 14216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:18.622002 14216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:49:18.627451 14216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:18.628568 14216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:18.629598 14216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:18.639174 14224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:18.639220 14216 server_base.cc:1034] running on GCE node
W20260924 04:49:18.639199 14222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:18.639215 14221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:18.639873 14216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:18.640512 14216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:18.641662 14216 hybrid_clock.cc:648] HybridClock initialized: now 1790225358641631 us; error 42 us; skew 500 ppm
Sep 24 04:49:18 dist-test-slave-hvcl krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:18.644335 14216 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:49:18.645221 14216 webserver.cc:460] Webserver started at http://127.0.98.1:45075/ using document root <none> and password file <none>
I20260924 04:49:18.645807 14216 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:18.645905 14216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:18.646113 14216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:49:18.648288 14216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "94d7c53d238242aa9d35668376b2a285"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "89d3cd90b31f7b82f1257f74321d6789"
server_key_iv: "b720bc1c04fa7b4781846a3f59a76b3a"
server_key_version: "encryptionkey@0"
I20260924 04:49:18.648844 14216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "94d7c53d238242aa9d35668376b2a285"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "89d3cd90b31f7b82f1257f74321d6789"
server_key_iv: "b720bc1c04fa7b4781846a3f59a76b3a"
server_key_version: "encryptionkey@0"
I20260924 04:49:18.652520 14216 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:49:18.654937 14231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:18.656224 14216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:49:18.656332 14216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "94d7c53d238242aa9d35668376b2a285"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "89d3cd90b31f7b82f1257f74321d6789"
server_key_iv: "b720bc1c04fa7b4781846a3f59a76b3a"
server_key_version: "encryptionkey@0"
I20260924 04:49:18.656435 14216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:18.672382 14216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:18.675498 14216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:18.675683 14216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:18.676268 14216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:18.677098 14216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:18.677145 14216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:18.677199 14216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:18.677228 14216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:18.685529 14216 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43849
I20260924 04:49:18.685564 14344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43849 every 8 connection(s)
I20260924 04:49:18.686429 14216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:49:18.689011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 14216
I20260924 04:49:18.689144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:49:18.689435 392 external_mini_cluster.cc:1428] Setting key a3f9e7ba993551a8db0f555e18374da3
Sep 24 04:49:18 dist-test-slave-hvcl krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:49:18.700413 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:18.688268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46851 (local address 127.0.98.62:33651)
0924 04:49:18.688412 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:18.688416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:18.689505 (+ 1089us) server_negotiation.cc:407] Connection header received
0924 04:49:18.690522 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:18.690526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:18.690573 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:18.690651 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:18.691895 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:18.692502 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:18.693789 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:18.694023 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:18.694114 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:18.696807 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:18.696810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:18.696812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:18.696839 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:18.698224 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:18.698741 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:18.698744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:18.698746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:18.698829 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:18.699200 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:18.699203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:18.699204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:18.699463 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:18.699552 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:18.700081 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 04:49:18.700234 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:49:18.701218 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:18.688635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:46851)
0924 04:49:18.689334 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:18.689400 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:18.690307 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:18.690833 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:18.690840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:18.691249 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:18.691744 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:18.691758 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:18.692624 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:18.692627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:18.693677 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:18.693688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:18.694173 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:18.694177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:18.694341 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:18.694920 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:18.694936 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:18.696633 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:18.698336 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:18.698343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:18.698367 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:18.698626 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:18.698909 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:18.698912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:18.698913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:18.699091 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:18.699564 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:18.699569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:18.699840 (+ 271us) client_negotiation.cc:769] Sending connection context
0924 04:49:18.699998 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 04:49:18.700283 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":112,"threads_started":1}
I20260924 04:49:18.702497 14345 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33651
I20260924 04:49:18.702725 14345 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:18.703248 14345 heartbeater.cc:511] Master 127.0.98.62:33651 requested a full tablet report, sending...
I20260924 04:49:18.704909 14151 ts_manager.cc:194] Registered new tserver with Master: 94d7c53d238242aa9d35668376b2a285 (127.0.98.1:43849)
I20260924 04:49:18.706522 14151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:49:18.719406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33651
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:18.830089 14350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:18.830374 14350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:18.830416 14350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:18.834483 14350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:18.834578 14350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:18.834661 14350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:49:18.840202 14350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:18.841277 14350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:18.842376 14350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:18.852178 14356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:18.852203 14355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:18.852174 14358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:18.852219 14350 server_base.cc:1034] running on GCE node
I20260924 04:49:18.852923 14350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:18.853626 14350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:18.854799 14350 hybrid_clock.cc:648] HybridClock initialized: now 1790225358854779 us; error 35 us; skew 500 ppm
Sep 24 04:49:18 dist-test-slave-hvcl krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:18.857722 14350 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:49:18.858660 14350 webserver.cc:460] Webserver started at http://127.0.98.2:34305/ using document root <none> and password file <none>
I20260924 04:49:18.859278 14350 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:18.859390 14350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:18.859608 14350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:49:18.861791 14350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "0a7f8b5c02a2405f85f86d7b3f377354"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "14be1e6ef492798b993ad7acdd8853c1"
server_key_iv: "1cd495e17c2bbcca30b1c73dca5b903b"
server_key_version: "encryptionkey@0"
I20260924 04:49:18.862339 14350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0a7f8b5c02a2405f85f86d7b3f377354"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "14be1e6ef492798b993ad7acdd8853c1"
server_key_iv: "1cd495e17c2bbcca30b1c73dca5b903b"
server_key_version: "encryptionkey@0"
I20260924 04:49:18.866277 14350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:49:18.868814 14365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:18.870162 14350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:49:18.870350 14350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "0a7f8b5c02a2405f85f86d7b3f377354"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "14be1e6ef492798b993ad7acdd8853c1"
server_key_iv: "1cd495e17c2bbcca30b1c73dca5b903b"
server_key_version: "encryptionkey@0"
I20260924 04:49:18.870465 14350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:18.887161 14350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:18.890733 14350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:18.890928 14350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:18.891573 14350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:18.892414 14350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:18.892462 14350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:18.892514 14350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:18.892541 14350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:18.901247 14350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42131
I20260924 04:49:18.901276 14478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42131 every 8 connection(s)
I20260924 04:49:18.902153 14350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:49:18.905980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 14350
I20260924 04:49:18.906095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:49:18.906409 392 external_mini_cluster.cc:1428] Setting key 3e943444deb853a1b310fd86f7a279eb
Sep 24 04:49:18 dist-test-slave-hvcl krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:49:18.917039 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:18.904061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42007 (local address 127.0.98.62:33651)
0924 04:49:18.904221 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:18.904225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:18.905351 (+ 1126us) server_negotiation.cc:407] Connection header received
0924 04:49:18.906551 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:18.906555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:18.906616 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:18.906694 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:18.908435 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:18.909206 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:18.910389 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:18.910616 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:18.910718 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:18.913481 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:18.913485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:18.913487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:18.913512 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:18.914851 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:18.915414 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:18.915417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:18.915418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:18.915497 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:18.915893 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:18.915898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:18.915899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:18.916131 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:18.916220 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:18.916775 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 04:49:18.916908 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 04:49:18.917929 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:18.904478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:42007)
0924 04:49:18.905154 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:18.905206 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:18.906177 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:18.907029 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:18.907039 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:18.907588 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:18.908287 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:18.908305 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:18.909350 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:18.909353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:18.910264 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:18.910273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:18.910776 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:18.910780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:18.911027 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:18.911642 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:18.911664 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:18.913283 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:18.915001 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:18.915007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:18.915026 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:18.915304 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:18.915587 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:18.915590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:18.915592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:18.915738 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:18.916227 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:18.916232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:18.916498 (+ 266us) client_negotiation.cc:769] Sending connection context
0924 04:49:18.916674 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 04:49:18.917011 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:49:18.919212 14479 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33651
I20260924 04:49:18.919467 14479 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:18.919986 14479 heartbeater.cc:511] Master 127.0.98.62:33651 requested a full tablet report, sending...
I20260924 04:49:18.921132 14151 ts_manager.cc:194] Registered new tserver with Master: 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131)
I20260924 04:49:18.921757 14151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:49:18.938249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33651
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:19.049660 14484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:19.049943 14484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:19.049988 14484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:19.054495 14484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:19.054589 14484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:19.054666 14484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:49:19.060217 14484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:19.061347 14484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:19.062407 14484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:19.072245 14489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:19.072332 14492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:19.072276 14490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:19.072424 14484 server_base.cc:1034] running on GCE node
I20260924 04:49:19.072911 14484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:19.073614 14484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:19.074836 14484 hybrid_clock.cc:648] HybridClock initialized: now 1790225359074803 us; error 52 us; skew 500 ppm
Sep 24 04:49:19 dist-test-slave-hvcl krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:19.077562 14484 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:49:19.078401 14484 webserver.cc:460] Webserver started at http://127.0.98.3:39853/ using document root <none> and password file <none>
I20260924 04:49:19.078914 14484 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:19.078995 14484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:19.079190 14484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:49:19.081164 14484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "8b3e207b1de6487fbcb1016eecd4b2e1"
format_stamp: "Formatted at 2026-09-24 04:49:19 on dist-test-slave-hvcl"
server_key: "e5323fe3e209623459ede60510fde0a9"
server_key_iv: "e4303e0586d48e53ed7f6c3197038f04"
server_key_version: "encryptionkey@0"
I20260924 04:49:19.081671 14484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8b3e207b1de6487fbcb1016eecd4b2e1"
format_stamp: "Formatted at 2026-09-24 04:49:19 on dist-test-slave-hvcl"
server_key: "e5323fe3e209623459ede60510fde0a9"
server_key_iv: "e4303e0586d48e53ed7f6c3197038f04"
server_key_version: "encryptionkey@0"
I20260924 04:49:19.085366 14484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 04:49:19.087767 14499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:19.089032 14484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:49:19.089133 14484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "8b3e207b1de6487fbcb1016eecd4b2e1"
format_stamp: "Formatted at 2026-09-24 04:49:19 on dist-test-slave-hvcl"
server_key: "e5323fe3e209623459ede60510fde0a9"
server_key_iv: "e4303e0586d48e53ed7f6c3197038f04"
server_key_version: "encryptionkey@0"
I20260924 04:49:19.089222 14484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:19.101737 14484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:19.104900 14484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:19.105057 14484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:19.105600 14484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:19.106453 14484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:19.106500 14484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:19.106540 14484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:19.106560 14484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:19.115170 14484 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43081
I20260924 04:49:19.115203 14612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43081 every 8 connection(s)
I20260924 04:49:19.116528 14484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:49:19.124631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 14484
I20260924 04:49:19.124747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:49:19.125007 392 external_mini_cluster.cc:1428] Setting key cf1815c9c823481e73c7cc2f3ad7ca83
Sep 24 04:49:19 dist-test-slave-hvcl krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:19.130806 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.118422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43809 (local address 127.0.98.62:33651)
0924 04:49:19.118578 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:19.118582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:19.119690 (+ 1108us) server_negotiation.cc:407] Connection header received
0924 04:49:19.120588 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:19.120592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:19.120641 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:19.120721 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:19.122036 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.122639 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.123949 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.124191 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.124282 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.127163 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:19.127167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:19.127170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:19.127202 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:19.128526 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:19.129085 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:19.129088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:19.129090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:19.129177 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:19.129576 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:19.129579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:19.129581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:19.129780 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:19.129883 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:19.130542 (+ 659us) server_negotiation.cc:299] Negotiation successful
0924 04:49:19.130661 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:49:19.131676 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.118825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:43809)
0924 04:49:19.119541 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:19.119584 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:19.120367 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:19.120939 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:19.120946 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:19.121364 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:19.121869 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.121885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.122768 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.122771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.123799 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.123812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.124400 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.124403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.124571 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.125238 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:19.125255 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:19.126954 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:19.128654 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:19.128660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:19.128673 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:19.128959 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:19.129293 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:19.129296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:19.129297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:19.129458 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:19.129927 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:19.129933 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:19.130208 (+ 275us) client_negotiation.cc:769] Sending connection context
0924 04:49:19.130444 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 04:49:19.130747 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":134,"threads_started":1}
I20260924 04:49:19.132929 14613 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33651
I20260924 04:49:19.133174 14613 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:19.133677 14613 heartbeater.cc:511] Master 127.0.98.62:33651 requested a full tablet report, sending...
I20260924 04:49:19.134752 14151 ts_manager.cc:194] Registered new tserver with Master: 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081)
I20260924 04:49:19.135432 14151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43809
I20260924 04:49:19.139164 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:49:19.148934 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.141122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.0.98.62:33651)
0924 04:49:19.141281 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:19.141286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:19.141422 (+ 136us) server_negotiation.cc:407] Connection header received
0924 04:49:19.141536 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:19.141539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:19.141583 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:19.141647 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:19.142393 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.143012 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.143826 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.144057 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.144141 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.145430 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:19.145434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:19.145436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:19.145462 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:19.146898 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:19.147455 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:19.147459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:19.147461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:19.147546 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:19.147837 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:19.147840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:19.147885 (+ 45us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:19.148195 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:19.148416 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:19.148656 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 04:49:19.148780 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 04:49:19.151726 14151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:49:19.154788 14151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:49:19.170184 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.163990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43849 (local address 127.0.0.1:54384)
0924 04:49:19.164877 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:19.164942 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:19.165081 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:19.165784 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:19.165790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:19.165813 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:19.165962 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.165969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.167218 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.167221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.168629 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.168640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.169779 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.169782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.169869 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.169888 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:49:19.169968 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:49:19.170026 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":24,"spinlock_wait_cycles":1920,"thread_start_us":82,"threads_started":1}
I20260924 04:49:19.170285 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.164662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.0.98.2:42131)
0924 04:49:19.165069 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:19.165081 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:19.165621 (+ 540us) server_negotiation.cc:407] Connection header received
0924 04:49:19.165698 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:19.165703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:19.165901 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:19.166040 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:19.166368 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.167495 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.168823 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.169441 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.169542 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.169705 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:19.170006 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 04:49:19.170117 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":124,"threads_started":1}
I20260924 04:49:19.170182 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.164355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42131 (local address 127.0.0.1:36560)
0924 04:49:19.165442 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:19.165456 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:19.165534 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:19.166149 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:19.166151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:19.166167 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:19.166252 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.166258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.167625 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.167629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.168638 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.168647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.169574 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.169578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.169705 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.169748 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 04:49:19.169860 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 04:49:19.170001 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"thread_start_us":105,"threads_started":1}
I20260924 04:49:19.170557 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.164841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.0.98.1:43849)
0924 04:49:19.165349 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:19.165362 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:19.165379 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:49:19.165440 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:19.165445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:19.165611 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:19.166111 (+ 500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:19.166262 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.167098 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.168777 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.169636 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.169775 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.169897 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:19.170271 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 04:49:19.170410 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":110,"threads_started":1}
I20260924 04:49:19.171126 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.164069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.0.98.3:43081)
0924 04:49:19.165258 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:19.165269 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:19.165293 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:49:19.165375 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:19.165380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:19.165611 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:19.165762 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:19.166147 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.167384 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.168840 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.169636 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.169751 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.169906 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:19.170828 (+ 922us) server_negotiation.cc:299] Negotiation successful
0924 04:49:19.170956 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":107,"threads_started":1}
I20260924 04:49:19.171388 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.163932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43081 (local address 127.0.0.1:55614)
0924 04:49:19.164878 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:19.164939 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:19.165080 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:19.165933 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:19.165936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:19.165949 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:19.166027 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.166033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.167530 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.167534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.168721 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.168729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.169969 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.169973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.170060 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.170083 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:49:19.170149 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:49:19.170220 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":70,"thread_start_us":134,"threads_started":1}
I20260924 04:49:19.173869 14547 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc2f68bc3ef4814bc28b95016f91346 (DEFAULT_TABLE table=test-table [id=9b367d14fd424ae58304e107d11c1f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:19.173868 14413 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc2f68bc3ef4814bc28b95016f91346 (DEFAULT_TABLE table=test-table [id=9b367d14fd424ae58304e107d11c1f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:19.173861 14279 tablet_service.cc:1467] Processing CreateTablet for tablet 7bc2f68bc3ef4814bc28b95016f91346 (DEFAULT_TABLE table=test-table [id=9b367d14fd424ae58304e107d11c1f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:19.175035 14279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc2f68bc3ef4814bc28b95016f91346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:19.175035 14413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc2f68bc3ef4814bc28b95016f91346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:19.175042 14547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bc2f68bc3ef4814bc28b95016f91346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:19.182279 14630 tablet_bootstrap.cc:492] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1: Bootstrap starting.
I20260924 04:49:19.183210 14631 tablet_bootstrap.cc:492] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354: Bootstrap starting.
I20260924 04:49:19.183245 14632 tablet_bootstrap.cc:492] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: Bootstrap starting.
I20260924 04:49:19.184381 14630 tablet_bootstrap.cc:654] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:19.185415 14632 tablet_bootstrap.cc:654] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:19.185415 14630 log.cc:824] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:19.185806 14631 tablet_bootstrap.cc:654] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:19.186625 14631 log.cc:824] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:19.186625 14632 log.cc:824] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:19.187542 14630 tablet_bootstrap.cc:492] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1: No bootstrap required, opened a new log
I20260924 04:49:19.187727 14630 ts_tablet_manager.cc:1397] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:49:19.188720 14632 tablet_bootstrap.cc:492] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: No bootstrap required, opened a new log
I20260924 04:49:19.188869 14632 ts_tablet_manager.cc:1397] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 04:49:19.188992 14631 tablet_bootstrap.cc:492] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354: No bootstrap required, opened a new log
I20260924 04:49:19.189153 14631 ts_tablet_manager.cc:1397] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260924 04:49:19.191610 14632 raft_consensus.cc:348] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } }
I20260924 04:49:19.191635 14630 raft_consensus.cc:348] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } }
I20260924 04:49:19.191841 14632 raft_consensus.cc:374] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:19.191895 14630 raft_consensus.cc:374] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:19.191923 14632 raft_consensus.cc:729] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d7c53d238242aa9d35668376b2a285, State: Initialized, Role: FOLLOWER
I20260924 04:49:19.191954 14630 raft_consensus.cc:729] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3e207b1de6487fbcb1016eecd4b2e1, State: Initialized, Role: FOLLOWER
I20260924 04:49:19.192369 14632 consensus_queue.cc:260] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } }
I20260924 04:49:19.192444 14630 consensus_queue.cc:260] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } }
I20260924 04:49:19.193141 14345 heartbeater.cc:503] Master 127.0.98.62:33651 was elected leader, sending a full tablet report...
I20260924 04:49:19.193117 14631 raft_consensus.cc:348] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } }
I20260924 04:49:19.193470 14632 ts_tablet_manager.cc:1428] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260924 04:49:19.193559 14630 ts_tablet_manager.cc:1428] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:49:19.193879 14613 heartbeater.cc:503] Master 127.0.98.62:33651 was elected leader, sending a full tablet report...
I20260924 04:49:19.193454 14631 raft_consensus.cc:374] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:19.193979 14631 raft_consensus.cc:729] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7f8b5c02a2405f85f86d7b3f377354, State: Initialized, Role: FOLLOWER
I20260924 04:49:19.194554 14631 consensus_queue.cc:260] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } }
I20260924 04:49:19.195214 14479 heartbeater.cc:503] Master 127.0.98.62:33651 was elected leader, sending a full tablet report...
I20260924 04:49:19.195518 14631 ts_tablet_manager.cc:1428] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260924 04:49:19.245357 14636 raft_consensus.cc:482] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:49:19.245553 14636 raft_consensus.cc:504] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } }
I20260924 04:49:19.246654 14636 leader_election.cc:288] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081), 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131)
I20260924 04:49:19.250801 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.247102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34467 (local address 127.0.98.3:43081)
0924 04:49:19.247230 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:19.247234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:19.247461 (+ 227us) server_negotiation.cc:407] Connection header received
0924 04:49:19.247592 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:19.247595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:19.247654 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:19.247726 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:19.248026 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.248576 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.249782 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.250185 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.250271 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.250335 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:19.250590 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 04:49:19.250659 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:49:19.250818 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.247089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42131 (local address 127.0.98.1:49543)
0924 04:49:19.247541 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:19.247555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:19.247654 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:19.247912 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:19.247914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:19.247929 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:19.248061 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.248068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.248838 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.248841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.249648 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.249658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.250334 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.250338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.250432 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.250498 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 04:49:19.250562 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:49:19.250652 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":35,"thread_start_us":132,"threads_started":1}
I20260924 04:49:19.250809 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.247295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49543 (local address 127.0.98.2:42131)
0924 04:49:19.247443 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:19.247447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:19.247643 (+ 196us) server_negotiation.cc:407] Connection header received
0924 04:49:19.247737 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:19.247740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:19.247821 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:19.247892 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:19.248169 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.248721 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.249781 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.250207 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.250311 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.250369 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:19.250598 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 04:49:19.250670 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:49:19.250821 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.246941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43081 (local address 127.0.98.1:34467)
0924 04:49:19.247336 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:19.247352 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:19.247477 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:19.247747 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:19.247750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:19.247768 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:19.247908 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.247915 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.248698 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.248702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.249641 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.249652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.250320 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.250324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.250421 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.250493 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 04:49:19.250554 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 04:49:19.250656 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":39,"thread_start_us":92,"threads_started":1}
I20260924 04:49:19.251503 14433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc2f68bc3ef4814bc28b95016f91346" candidate_uuid: "94d7c53d238242aa9d35668376b2a285" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" is_pre_election: true
I20260924 04:49:19.251832 14433 raft_consensus.cc:2454] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d7c53d238242aa9d35668376b2a285 in term 0.
I20260924 04:49:19.251861 14567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc2f68bc3ef4814bc28b95016f91346" candidate_uuid: "94d7c53d238242aa9d35668376b2a285" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" is_pre_election: true
I20260924 04:49:19.252120 14567 raft_consensus.cc:2454] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d7c53d238242aa9d35668376b2a285 in term 0.
I20260924 04:49:19.252398 14232 leader_election.cc:302] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7f8b5c02a2405f85f86d7b3f377354, 94d7c53d238242aa9d35668376b2a285; no voters:
I20260924 04:49:19.252667 14636 raft_consensus.cc:2781] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:49:19.252758 14636 raft_consensus.cc:482] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:49:19.252810 14636 raft_consensus.cc:3037] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:19.253826 14636 raft_consensus.cc:504] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } }
I20260924 04:49:19.254146 14636 leader_election.cc:288] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 1 election: Requested vote from peers 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081), 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131)
I20260924 04:49:19.254520 14567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc2f68bc3ef4814bc28b95016f91346" candidate_uuid: "94d7c53d238242aa9d35668376b2a285" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1"
I20260924 04:49:19.254631 14567 raft_consensus.cc:3037] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:19.254665 14433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bc2f68bc3ef4814bc28b95016f91346" candidate_uuid: "94d7c53d238242aa9d35668376b2a285" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a7f8b5c02a2405f85f86d7b3f377354"
I20260924 04:49:19.254830 14433 raft_consensus.cc:3037] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:19.255759 14567 raft_consensus.cc:2454] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d7c53d238242aa9d35668376b2a285 in term 1.
I20260924 04:49:19.255934 14433 raft_consensus.cc:2454] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d7c53d238242aa9d35668376b2a285 in term 1.
I20260924 04:49:19.256114 14234 leader_election.cc:302] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b3e207b1de6487fbcb1016eecd4b2e1, 94d7c53d238242aa9d35668376b2a285; no voters:
I20260924 04:49:19.256301 14636 raft_consensus.cc:2781] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:49:19.256537 14636 raft_consensus.cc:686] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 1 LEADER]: Becoming Leader. State: Replica: 94d7c53d238242aa9d35668376b2a285, State: Running, Role: LEADER
I20260924 04:49:19.256913 14636 consensus_queue.cc:237] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } }
I20260924 04:49:19.260145 14151 catalog_manager.cc:5495] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94d7c53d238242aa9d35668376b2a285 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "94d7c53d238242aa9d35668376b2a285" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } health_report { overall_health: UNKNOWN } } }
I20260924 04:49:19.295583 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.291813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.0.98.1:43849)
0924 04:49:19.291987 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:19.291990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:19.292003 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:49:19.292097 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:19.292101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:19.292161 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:19.292266 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:49:19.292505 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.293110 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.294041 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.294286 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.294386 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.294738 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:49:19.295230 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:49:19.295337 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:19.295406 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 04:49:19.295463 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:49:19.301541 14567 raft_consensus.cc:1261] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 1 FOLLOWER]: Refusing update from remote peer 94d7c53d238242aa9d35668376b2a285: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:49:19.301554 14433 raft_consensus.cc:1261] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354 [term 1 FOLLOWER]: Refusing update from remote peer 94d7c53d238242aa9d35668376b2a285: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:49:19.302302 14641 consensus_queue.cc:1035] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:49:19.302484 14636 consensus_queue.cc:1035] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:49:19.309738 14646 mvcc.cc:205] Tried to move back new op lower bound from 7332763071691792384 to 7332763071519031296. Current Snapshot: MvccSnapshot[applied={T|T < 7332763071691792384}]
I20260924 04:49:19.318378 14153 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:49:19.318586 14153 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:49:19.321358 14153 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Sending DeleteTablet for 3 replicas of tablet 7bc2f68bc3ef4814bc28b95016f91346
I20260924 04:49:19.322237 14279 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc2f68bc3ef4814bc28b95016f91346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54384
I20260924 04:49:19.322227 14547 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc2f68bc3ef4814bc28b95016f91346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55614
I20260924 04:49:19.322227 14413 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc2f68bc3ef4814bc28b95016f91346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36560
I20260924 04:49:19.322790 14654 tablet_replica.cc:330] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1: stopping tablet replica
I20260924 04:49:19.322791 14653 tablet_replica.cc:330] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: stopping tablet replica
I20260924 04:49:19.323186 14653 raft_consensus.cc:2229] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:49:19.323221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 14216
I20260924 04:49:19.323226 14654 raft_consensus.cc:2229] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:49:19.324330 14655 tablet_replica.cc:330] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354: stopping tablet replica
I20260924 04:49:19.324326 14654 raft_consensus.cc:2258] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:49:19.324681 14655 raft_consensus.cc:2229] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:49:19.324990 14655 raft_consensus.cc:2258] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:49:19.326442 14654 ts_tablet_manager.cc:1905] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:49:19.326442 14655 ts_tablet_manager.cc:1905] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:49:19.329228 14655 ts_tablet_manager.cc:1918] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:49:19.329228 14654 ts_tablet_manager.cc:1918] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:49:19.329329 14654 log.cc:1196] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/7bc2f68bc3ef4814bc28b95016f91346
I20260924 04:49:19.329329 14655 log.cc:1196] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/7bc2f68bc3ef4814bc28b95016f91346
W20260924 04:49:19.329476 14141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43849 (error 108)
I20260924 04:49:19.329646 14654 ts_tablet_manager.cc:1939] T 7bc2f68bc3ef4814bc28b95016f91346 P 8b3e207b1de6487fbcb1016eecd4b2e1: Deleting consensus metadata
I20260924 04:49:19.329646 14655 ts_tablet_manager.cc:1939] T 7bc2f68bc3ef4814bc28b95016f91346 P 0a7f8b5c02a2405f85f86d7b3f377354: Deleting consensus metadata
I20260924 04:49:19.330550 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.330217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43849 (local address 127.0.0.1:54414)
0924 04:49:19.330391 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:19.330459 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 04:49:19.330622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 14350
W20260924 04:49:19.330830 14141 catalog_manager.cc:4584] TS 94d7c53d238242aa9d35668376b2a285 (127.0.98.1:43849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bc2f68bc3ef4814bc28b95016f91346: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43849: connect: Connection refused (error 111)
I20260924 04:49:19.330921 14140 catalog_manager.cc:4841] TS 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081): tablet 7bc2f68bc3ef4814bc28b95016f91346 (table test-table [id=9b367d14fd424ae58304e107d11c1f28]) successfully deleted
I20260924 04:49:19.335817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 14484
I20260924 04:49:19.336738 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.336011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42131 (local address 127.0.0.1:36572)
0924 04:49:19.336586 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:19.336645 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483}
I20260924 04:49:19.341114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 14122
I20260924 04:49:19.346889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33651
--webserver_interface=127.0.98.62
--webserver_port=45239
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:19.456709 14657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:19.457010 14657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:19.457065 14657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:19.461154 14657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:49:19.461231 14657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:19.461257 14657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:19.461279 14657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:49:19.461298 14657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:49:19.466714 14657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:19.467878 14657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:19.468936 14657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:49:19.482090 14657 server_base.cc:1034] running on GCE node
W20260924 04:49:19.482085 14664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:19.482106 14663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:19.482074 14666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:19.483047 14657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:19.484354 14657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:19.485541 14657 hybrid_clock.cc:648] HybridClock initialized: now 1790225359485510 us; error 41 us; skew 500 ppm
Sep 24 04:49:19 dist-test-slave-hvcl krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:19.488451 14657 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:49:19.489393 14657 webserver.cc:460] Webserver started at http://127.0.98.62:45239/ using document root <none> and password file <none>
I20260924 04:49:19.489979 14657 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:19.490054 14657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:19.494014 14657 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:49:19.496330 14673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:19.497627 14657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:49:19.497738 14657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "64d60a82d7e74b81b1e2739c1140b868"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "082b86a05e9dc0a37573877508db8cb4"
server_key_iv: "7c9b4ab5bcb7c54780f61f558c15a02c"
server_key_version: "encryptionkey@0"
I20260924 04:49:19.498351 14657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:19.510639 14657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:19.513855 14657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:19.514014 14657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:19.520838 14657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33651
I20260924 04:49:19.521132 14725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33651 every 8 connection(s)
I20260924 04:49:19.521973 14657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:49:19.523227 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 14657
I20260924 04:49:19.523641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43849
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33651
--webserver_port=45075
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:49:19.527176 14726 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:49:19.528911 14726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Bootstrap starting.
I20260924 04:49:19.547797 14726 log.cc:824] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:19.556794 14726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:49:19.557165 14726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Bootstrap complete.
I20260924 04:49:19.561065 14726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } }
I20260924 04:49:19.561477 14726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d60a82d7e74b81b1e2739c1140b868, State: Initialized, Role: FOLLOWER
I20260924 04:49:19.561992 14726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } }
I20260924 04:49:19.562124 14726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:49:19.562199 14726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:49:19.562304 14726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:49:19.564273 14726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } }
I20260924 04:49:19.564606 14726 leader_election.cc:302] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64d60a82d7e74b81b1e2739c1140b868; no voters:
I20260924 04:49:19.564910 14726 leader_election.cc:288] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:49:19.565037 14730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:49:19.565243 14730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 2 LEADER]: Becoming Leader. State: Replica: 64d60a82d7e74b81b1e2739c1140b868, State: Running, Role: LEADER
I20260924 04:49:19.565539 14730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } }
I20260924 04:49:19.566203 14726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:49:19.566896 14731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "64d60a82d7e74b81b1e2739c1140b868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } } }
I20260924 04:49:19.567052 14731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:19.567363 14732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64d60a82d7e74b81b1e2739c1140b868. Latest consensus state: current_term: 2 leader_uuid: "64d60a82d7e74b81b1e2739c1140b868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } } }
I20260924 04:49:19.567443 14732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:19.567870 14737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:49:19.572561 14737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:49:19.573213 14737 catalog_manager.cc:1259] Loaded cluster ID: 6e2db1f69d4d43a48e50e048afe00f74
I20260924 04:49:19.573274 14737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:49:19.574365 14737 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 04:49:19.577165 14737 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 04:49:19.581573 14737 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:49:19.582326 14737 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Loaded TSK: 0
I20260924 04:49:19.583356 14737 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:49:19.663251 14728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:19.663532 14728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:19.663576 14728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:19.667897 14728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:19.667969 14728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:19.668037 14728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:49:19.673554 14728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45075
--tserver_master_addrs=127.0.98.62:33651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:19.674685 14728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:19.675755 14728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:19.686637 14756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:19.686666 14753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:19.687000 14728 server_base.cc:1034] running on GCE node
W20260924 04:49:19.686635 14754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:19.687517 14728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:19.688216 14728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:19.689407 14728 hybrid_clock.cc:648] HybridClock initialized: now 1790225359689378 us; error 51 us; skew 500 ppm
Sep 24 04:49:19 dist-test-slave-hvcl krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:19.692340 14728 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:49:19.693359 14728 webserver.cc:460] Webserver started at http://127.0.98.1:45075/ using document root <none> and password file <none>
I20260924 04:49:19.693933 14728 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:19.694007 14728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:19.705338 14728 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.002s sys 0.000s
I20260924 04:49:19.708564 14763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:19.710357 14728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:49:19.710515 14728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "94d7c53d238242aa9d35668376b2a285"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "89d3cd90b31f7b82f1257f74321d6789"
server_key_iv: "b720bc1c04fa7b4781846a3f59a76b3a"
server_key_version: "encryptionkey@0"
I20260924 04:49:19.711349 14728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:19.736481 14728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:19.740592 14728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:19.740777 14728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:19.741443 14728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:19.742640 14770 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:49:19.746876 14728 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:49:19.746958 14728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260924 04:49:19.747045 14728 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:49:19.749205 14728 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:49:19.749264 14728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:49:19.749567 14770 tablet_bootstrap.cc:492] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: Bootstrap starting.
I20260924 04:49:19.757289 14728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43849
I20260924 04:49:19.757645 14877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43849 every 8 connection(s)
I20260924 04:49:19.758288 14728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:49:19.765251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 14728
I20260924 04:49:19.765837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42131
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33651
--webserver_port=34305
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:49:19.769879 14770 log.cc:824] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:49:19 dist-test-slave-hvcl krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:19.778115 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.760451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:40717)
0924 04:49:19.761039 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:19.761098 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:19.762196 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:19.763454 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:19.763464 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:19.764030 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:19.764830 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.764851 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.766702 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.766706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.768326 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.768338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.769308 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.769312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.769524 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.770213 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:19.770236 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:19.771969 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:19.774599 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:19.774604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:19.774615 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:19.774958 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:19.775410 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:19.775413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:19.775415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:19.775584 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:19.776355 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:19.776362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:19.776653 (+ 291us) client_negotiation.cc:769] Sending connection context
0924 04:49:19.776837 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 04:49:19.777167 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":152,"threads_started":1}
I20260924 04:49:19.778117 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.760574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40717 (local address 127.0.98.62:33651)
0924 04:49:19.761065 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:19.761084 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:19.761219 (+ 135us) server_negotiation.cc:407] Connection header received
0924 04:49:19.762652 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:19.762663 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:19.763016 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:19.763306 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:19.765055 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.766537 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.768456 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.769152 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.769314 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.772209 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:19.772217 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:19.772247 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:19.772285 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:19.774456 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:19.775107 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:19.775114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:19.775120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:19.775298 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:19.775696 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:19.775699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:19.775701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:19.776213 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:19.776359 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:19.777002 (+ 643us) server_negotiation.cc:299] Negotiation successful
0924 04:49:19.777307 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":123,"threads_started":1}
I20260924 04:49:19.778383 14770 tablet_bootstrap.cc:492] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:49:19.778812 14770 tablet_bootstrap.cc:492] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: Bootstrap complete.
I20260924 04:49:19.779784 14770 ts_tablet_manager.cc:1397] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.008s
I20260924 04:49:19.781152 14878 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33651
I20260924 04:49:19.781404 14878 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:19.781972 14878 heartbeater.cc:511] Master 127.0.98.62:33651 requested a full tablet report, sending...
I20260924 04:49:19.782469 14770 raft_consensus.cc:348] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } }
I20260924 04:49:19.782904 14770 raft_consensus.cc:729] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d7c53d238242aa9d35668376b2a285, State: Initialized, Role: FOLLOWER
I20260924 04:49:19.783423 14770 consensus_queue.cc:260] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } } peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } }
I20260924 04:49:19.784473 14770 ts_tablet_manager.cc:1428] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:49:19.784904 14690 ts_manager.cc:194] Registered new tserver with Master: 94d7c53d238242aa9d35668376b2a285 (127.0.98.1:43849)
I20260924 04:49:19.788470 14690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40717
I20260924 04:49:19.791143 14878 heartbeater.cc:503] Master 127.0.98.62:33651 was elected leader, sending a full tablet report...
Sep 24 04:49:19 dist-test-slave-hvcl krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:49:19.798115 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.788045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.0.98.1:43849)
0924 04:49:19.788837 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:19.788849 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:19.788873 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:49:19.788947 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:19.788952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:19.789146 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:19.789275 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:19.789983 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.790710 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.791738 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.792239 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.792345 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.794703 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:19.794711 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:19.794721 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:19.794748 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:19.796339 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:19.796794 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:19.796799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:19.796800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:19.796877 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:19.797134 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:19.797137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:19.797139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:19.797378 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:19.797494 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:19.797825 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 04:49:19.797976 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":108,"threads_started":1}
I20260924 04:49:19.798115 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.787947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43849 (local address 127.0.0.1:54428)
0924 04:49:19.788537 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:19.788582 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:19.788711 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:19.789341 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:19.789347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:19.789729 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:19.789848 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.789856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.790834 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.790837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.791620 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.791633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.792353 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.792374 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.792490 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.793095 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:19.793110 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:19.794564 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:19.796454 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:19.796462 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:19.796467 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:19.796683 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:19.796965 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:19.796968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:19.796970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:19.797052 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:19.797489 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:19.797495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:19.797613 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 04:49:19.797730 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 04:49:19.797935 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":116,"threads_started":1}
I20260924 04:49:19.799153 14812 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bc2f68bc3ef4814bc28b95016f91346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54428
I20260924 04:49:19.799983 14887 tablet_replica.cc:330] stopping tablet replica
I20260924 04:49:19.800294 14887 raft_consensus.cc:2229] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:49:19.800561 14887 raft_consensus.cc:2258] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:49:19.801731 14887 ts_tablet_manager.cc:1905] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:49:19.806239 14887 ts_tablet_manager.cc:1918] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:49:19.806348 14887 log.cc:1196] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal/wals/7bc2f68bc3ef4814bc28b95016f91346
I20260924 04:49:19.806627 14887 ts_tablet_manager.cc:1939] T 7bc2f68bc3ef4814bc28b95016f91346 P 94d7c53d238242aa9d35668376b2a285: Deleting consensus metadata
I20260924 04:49:19.808700 14677 catalog_manager.cc:4841] TS 94d7c53d238242aa9d35668376b2a285 (127.0.98.1:43849): tablet 7bc2f68bc3ef4814bc28b95016f91346 (table test-table [id=9b367d14fd424ae58304e107d11c1f28]) successfully deleted
W20260924 04:49:19.884521 14882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:19.884773 14882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:19.884814 14882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:19.889302 14882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:19.889375 14882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:19.889437 14882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:49:19.894786 14882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34305
--tserver_master_addrs=127.0.98.62:33651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:19.895875 14882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:19.897109 14882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:19.908068 14893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:19.908088 14895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:19.908071 14892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:19.908584 14882 server_base.cc:1034] running on GCE node
I20260924 04:49:19.908972 14882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:19.909726 14882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:19.910926 14882 hybrid_clock.cc:648] HybridClock initialized: now 1790225359910862 us; error 87 us; skew 500 ppm
Sep 24 04:49:19 dist-test-slave-hvcl krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:19.914011 14882 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:49:19.915045 14882 webserver.cc:460] Webserver started at http://127.0.98.2:34305/ using document root <none> and password file <none>
I20260924 04:49:19.915613 14882 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:19.915693 14882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:19.919692 14882 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:49:19.921856 14902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:19.923436 14882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:49:19.923558 14882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "0a7f8b5c02a2405f85f86d7b3f377354"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "14be1e6ef492798b993ad7acdd8853c1"
server_key_iv: "1cd495e17c2bbcca30b1c73dca5b903b"
server_key_version: "encryptionkey@0"
I20260924 04:49:19.924127 14882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:19.938095 14882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:19.941551 14882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:19.941730 14882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:19.942332 14882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:19.943195 14882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:19.943243 14882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:19.943341 14882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:19.943370 14882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:19.952314 14882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42131
I20260924 04:49:19.952353 15015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42131 every 8 connection(s)
I20260924 04:49:19.953373 14882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 04:49:19 dist-test-slave-hvcl krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:19.963008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 14882
I20260924 04:49:19.963475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43081
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33651
--webserver_port=39853
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:49:19.967613 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.955199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39785 (local address 127.0.98.62:33651)
0924 04:49:19.955339 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:19.955342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:19.956437 (+ 1095us) server_negotiation.cc:407] Connection header received
0924 04:49:19.957317 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:19.957321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:19.957383 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:19.957463 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:19.958679 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.959262 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.960401 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.960632 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.960720 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.963415 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:19.963419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:19.963422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:19.963455 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:19.965016 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:19.965772 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:19.965775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:19.965777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:19.965877 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:19.966265 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:19.966268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:19.966270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:19.966476 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:19.966566 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:19.967269 (+ 703us) server_negotiation.cc:299] Negotiation successful
0924 04:49:19.967432 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:49:19.968477 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:19.955583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:39785)
0924 04:49:19.956235 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:19.956277 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:19.957107 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:19.957655 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:19.957662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:19.958036 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:19.958528 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.958541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.959382 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.959385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.960270 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:19.960280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:19.960781 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:19.960785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:19.960957 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:19.961540 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:19.961557 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:19.963145 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:19.965163 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:19.965171 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:19.965191 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:19.965639 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:19.965977 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:19.965980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:19.965982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:19.966172 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:19.966605 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:19.966612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:19.966877 (+ 265us) client_negotiation.cc:769] Sending connection context
0924 04:49:19.967142 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 04:49:19.967480 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":103,"threads_started":1}
I20260924 04:49:19.969753 15016 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33651
I20260924 04:49:19.970016 15016 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:19.970563 15016 heartbeater.cc:511] Master 127.0.98.62:33651 requested a full tablet report, sending...
I20260924 04:49:19.971711 14690 ts_manager.cc:194] Registered new tserver with Master: 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131)
I20260924 04:49:19.972380 14690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39785
W20260924 04:49:20.075354 15019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:20.075649 15019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:20.075695 15019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:20.079926 15019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:20.079994 15019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:20.080054 15019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:49:20.085744 15019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39853
--tserver_master_addrs=127.0.98.62:33651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:20.086844 15019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:20.087909 15019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:20.097358 15024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:20.097514 15019 server_base.cc:1034] running on GCE node
W20260924 04:49:20.097376 15027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:20.097447 15025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:20.098006 15019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:20.098635 15019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:20.099792 15019 hybrid_clock.cc:648] HybridClock initialized: now 1790225360099771 us; error 31 us; skew 500 ppm
Sep 24 04:49:20 dist-test-slave-hvcl krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:20.102599 15019 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:49:20.103511 15019 webserver.cc:460] Webserver started at http://127.0.98.3:39853/ using document root <none> and password file <none>
I20260924 04:49:20.104027 15019 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:20.104101 15019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:20.108039 15019 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:49:20.110194 15034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:20.111547 15019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:49:20.111650 15019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "8b3e207b1de6487fbcb1016eecd4b2e1"
format_stamp: "Formatted at 2026-09-24 04:49:19 on dist-test-slave-hvcl"
server_key: "e5323fe3e209623459ede60510fde0a9"
server_key_iv: "e4303e0586d48e53ed7f6c3197038f04"
server_key_version: "encryptionkey@0"
I20260924 04:49:20.112159 15019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:20.138738 15019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:20.142380 15019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:20.142537 15019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:20.143101 15019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:20.143935 15019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:20.143985 15019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:20.144024 15019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:20.144038 15019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:20.152599 15019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43081
I20260924 04:49:20.152634 15147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43081 every 8 connection(s)
I20260924 04:49:20.153618 15019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:49:20.160817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 15019
Sep 24 04:49:20 dist-test-slave-hvcl krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:20.167555 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.155405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48025 (local address 127.0.98.62:33651)
0924 04:49:20.155537 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:20.155540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:20.156580 (+ 1040us) server_negotiation.cc:407] Connection header received
0924 04:49:20.157481 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:20.157484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:20.157547 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:20.157618 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:20.158819 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.159403 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.160559 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.160785 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.160875 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:20.163585 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:20.163589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:20.163591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:20.163619 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:20.165058 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:20.165613 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:20.165617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:20.165618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:20.165716 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:20.166148 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:20.166152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:20.166153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:20.166354 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:20.166441 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:20.167172 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 04:49:20.167381 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:49:20.168598 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.155784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:48025)
0924 04:49:20.156421 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:20.156462 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:20.157261 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:20.157818 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:20.157825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:20.158193 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:20.158668 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:20.158681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.159524 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.159528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:20.160431 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:20.160440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.160935 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.160939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:20.161096 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:20.161723 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:20.161743 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:20.163419 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:20.165166 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:20.165174 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:20.165192 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:20.165487 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:20.165812 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:20.165816 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:20.165818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:20.166010 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:20.166486 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:20.166493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:20.166856 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 04:49:20.167111 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 04:49:20.167544 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":108,"threads_started":1}
I20260924 04:49:20.169726 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.161582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.0.98.62:33651)
0924 04:49:20.161882 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:20.161885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:20.162072 (+ 187us) server_negotiation.cc:407] Connection header received
0924 04:49:20.162235 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:20.162238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:20.162284 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:20.162429 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:20.163167 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.163764 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.164639 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.164869 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.164957 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:20.166433 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:20.166437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:20.166440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:20.166466 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:20.168070 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:20.168552 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:20.168555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:20.168557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:20.168646 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:20.168927 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:20.168931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:20.168933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:20.169113 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:20.169341 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:20.169460 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 04:49:20.169571 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
I20260924 04:49:20.169796 15148 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33651
I20260924 04:49:20.170037 15148 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:20.170568 15148 heartbeater.cc:511] Master 127.0.98.62:33651 requested a full tablet report, sending...
I20260924 04:49:20.171568 14690 ts_manager.cc:194] Registered new tserver with Master: 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081)
I20260924 04:49:20.172081 14690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48025
I20260924 04:49:20.183452 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:49:20.187877 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.184157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.0.98.62:33651)
0924 04:49:20.184328 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:20.184331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:20.184345 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:49:20.184576 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:20.184579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:20.184685 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:20.184787 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:49:20.185033 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.185584 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.186491 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.186724 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.186805 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:20.187122 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:49:20.187543 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:49:20.187633 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:20.187705 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 04:49:20.187761 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:49:20.188548 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:49:20.191798 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:49:20.199720 14812 tablet_service.cc:1467] Processing CreateTablet for tablet 5d91192d500a48ea990353f87044a58e (DEFAULT_TABLE table=test-table [id=e88198644b1e4649adb4042e1a11cd12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:20.200415 14812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d91192d500a48ea990353f87044a58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:20.203083 14770 tablet_bootstrap.cc:492] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285: Bootstrap starting.
I20260924 04:49:20.204013 14770 tablet_bootstrap.cc:654] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:20.204485 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.198705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42131 (local address 127.0.0.1:36580)
0924 04:49:20.198857 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:20.198872 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:20.198943 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:20.200057 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:20.200060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:20.200079 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:20.200249 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:20.200261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.201691 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.201694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:20.202950 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:20.202960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.204053 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.204057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:20.204151 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:20.204197 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:49:20.204257 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:49:20.204348 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":39}
I20260924 04:49:20.204833 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.198935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.0.98.2:42131)
0924 04:49:20.199641 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:20.199650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:20.199667 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:49:20.199732 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:20.199737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:20.199893 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:20.200272 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:20.200423 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.201583 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.203124 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.203932 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.204031 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:20.204386 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:20.204439 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 04:49:20.204712 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":118,"threads_started":1}
I20260924 04:49:20.204895 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.198781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43081 (local address 127.0.0.1:55622)
0924 04:49:20.199152 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:20.199164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:20.199235 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:20.200092 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:20.200094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:20.200108 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:20.200229 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:20.200235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.201741 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.201744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:20.203099 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:20.203110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.204488 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.204493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:20.204581 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:20.204604 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:49:20.204675 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:49:20.204745 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":3,"spinlock_wait_cycles":2304,"thread_start_us":111,"threads_started":1}
I20260924 04:49:20.205459 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.199216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.0.98.3:43081)
0924 04:49:20.199556 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:20.199568 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:20.199592 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:49:20.199678 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:20.199683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:20.199916 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:20.200074 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:20.200351 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.201574 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.203329 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.204128 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.204247 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:20.204421 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:20.205203 (+ 782us) server_negotiation.cc:299] Negotiation successful
0924 04:49:20.205316 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":127,"threads_started":1}
I20260924 04:49:20.205987 14770 tablet_bootstrap.cc:492] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285: No bootstrap required, opened a new log
I20260924 04:49:20.206071 14770 ts_tablet_manager.cc:1397] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 04:49:20.206516 14770 raft_consensus.cc:348] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:20.206645 14770 raft_consensus.cc:374] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:20.206682 14770 raft_consensus.cc:729] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d7c53d238242aa9d35668376b2a285, State: Initialized, Role: FOLLOWER
I20260924 04:49:20.206805 14770 consensus_queue.cc:260] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:20.207172 14770 ts_tablet_manager.cc:1428] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:49:20.208060 14950 tablet_service.cc:1467] Processing CreateTablet for tablet 5d91192d500a48ea990353f87044a58e (DEFAULT_TABLE table=test-table [id=e88198644b1e4649adb4042e1a11cd12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:20.208359 15082 tablet_service.cc:1467] Processing CreateTablet for tablet 5d91192d500a48ea990353f87044a58e (DEFAULT_TABLE table=test-table [id=e88198644b1e4649adb4042e1a11cd12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:20.209096 14950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d91192d500a48ea990353f87044a58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:20.209388 15082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d91192d500a48ea990353f87044a58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:20.216219 15161 tablet_bootstrap.cc:492] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1: Bootstrap starting.
I20260924 04:49:20.216825 15162 tablet_bootstrap.cc:492] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354: Bootstrap starting.
I20260924 04:49:20.218858 15161 tablet_bootstrap.cc:654] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:20.219318 15162 tablet_bootstrap.cc:654] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:20.220202 15162 log.cc:824] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:20.220202 15161 log.cc:824] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:20.222191 15162 tablet_bootstrap.cc:492] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354: No bootstrap required, opened a new log
I20260924 04:49:20.222211 15161 tablet_bootstrap.cc:492] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1: No bootstrap required, opened a new log
I20260924 04:49:20.222394 15161 ts_tablet_manager.cc:1397] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:49:20.222391 15162 ts_tablet_manager.cc:1397] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:49:20.225664 15162 raft_consensus.cc:348] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:20.225667 15161 raft_consensus.cc:348] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:20.225924 15162 raft_consensus.cc:374] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:20.225926 15161 raft_consensus.cc:374] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:20.225960 15162 raft_consensus.cc:729] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7f8b5c02a2405f85f86d7b3f377354, State: Initialized, Role: FOLLOWER
I20260924 04:49:20.225975 15161 raft_consensus.cc:729] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3e207b1de6487fbcb1016eecd4b2e1, State: Initialized, Role: FOLLOWER
I20260924 04:49:20.226364 15162 consensus_queue.cc:260] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:20.226365 15161 consensus_queue.cc:260] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:20.227051 15148 heartbeater.cc:503] Master 127.0.98.62:33651 was elected leader, sending a full tablet report...
I20260924 04:49:20.227362 15162 ts_tablet_manager.cc:1428] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:49:20.227366 15161 ts_tablet_manager.cc:1428] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:49:20.228065 15016 heartbeater.cc:503] Master 127.0.98.62:33651 was elected leader, sending a full tablet report...
W20260924 04:49:20.259523 14879 tablet.cc:2369] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:49:20.315801 14884 raft_consensus.cc:482] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:49:20.316035 14884 raft_consensus.cc:504] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:20.317199 14884 leader_election.cc:288] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131), 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081)
I20260924 04:49:20.321137 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.317586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58225 (local address 127.0.98.3:43081)
0924 04:49:20.317823 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:20.317828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:20.317847 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:49:20.318032 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:20.318035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:20.318090 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:20.318159 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:20.318428 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.318951 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.320121 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.320536 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.320627 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:20.320686 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:20.320929 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 04:49:20.320999 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260924 04:49:20.321142 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.317586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57503 (local address 127.0.98.2:42131)
0924 04:49:20.317713 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:20.317717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:20.317933 (+ 216us) server_negotiation.cc:407] Connection header received
0924 04:49:20.318032 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:20.318035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:20.318077 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:20.318133 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:20.318433 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.318949 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.320127 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.320521 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.320617 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:20.320682 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:20.320936 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 04:49:20.321007 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:49:20.321177 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.317474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42131 (local address 127.0.98.1:57503)
0924 04:49:20.317819 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:20.317839 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:20.317945 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:20.318165 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:20.318167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:20.318182 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:20.318322 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:20.318327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.319063 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.319066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:20.319998 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:20.320007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.320633 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.320637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:20.320748 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:20.320835 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 04:49:20.320899 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:49:20.321015 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":138,"spinlock_wait_cycles":896,"thread_start_us":72,"threads_started":1}
I20260924 04:49:20.321189 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.317474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43081 (local address 127.0.98.1:58225)
0924 04:49:20.317776 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:20.317790 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:20.317895 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:20.318181 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:20.318183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:20.318222 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:20.318318 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:20.318324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.319080 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.319084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:20.320001 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:20.320011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.320655 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.320659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:20.320780 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:20.320834 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 04:49:20.320898 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:49:20.321019 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":73,"spinlock_wait_cycles":28032,"thread_start_us":99,"threads_started":1}
I20260924 04:49:20.322005 15102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d91192d500a48ea990353f87044a58e" candidate_uuid: "94d7c53d238242aa9d35668376b2a285" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" is_pre_election: true
I20260924 04:49:20.322006 14970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d91192d500a48ea990353f87044a58e" candidate_uuid: "94d7c53d238242aa9d35668376b2a285" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" is_pre_election: true
I20260924 04:49:20.322331 15102 raft_consensus.cc:2454] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d7c53d238242aa9d35668376b2a285 in term 0.
I20260924 04:49:20.322331 14970 raft_consensus.cc:2454] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d7c53d238242aa9d35668376b2a285 in term 0.
I20260924 04:49:20.322794 14766 leader_election.cc:302] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b3e207b1de6487fbcb1016eecd4b2e1, 94d7c53d238242aa9d35668376b2a285; no voters:
I20260924 04:49:20.323055 14884 raft_consensus.cc:2781] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:49:20.323136 14884 raft_consensus.cc:482] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:49:20.323187 14884 raft_consensus.cc:3037] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:20.324354 14884 raft_consensus.cc:504] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:20.324710 14884 leader_election.cc:288] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 1 election: Requested vote from peers 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131), 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081)
I20260924 04:49:20.325078 14970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d91192d500a48ea990353f87044a58e" candidate_uuid: "94d7c53d238242aa9d35668376b2a285" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a7f8b5c02a2405f85f86d7b3f377354"
I20260924 04:49:20.325078 15102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d91192d500a48ea990353f87044a58e" candidate_uuid: "94d7c53d238242aa9d35668376b2a285" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1"
I20260924 04:49:20.325219 14970 raft_consensus.cc:3037] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:20.325219 15102 raft_consensus.cc:3037] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:20.326540 15102 raft_consensus.cc:2454] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d7c53d238242aa9d35668376b2a285 in term 1.
I20260924 04:49:20.326540 14970 raft_consensus.cc:2454] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d7c53d238242aa9d35668376b2a285 in term 1.
I20260924 04:49:20.326829 14766 leader_election.cc:302] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b3e207b1de6487fbcb1016eecd4b2e1, 94d7c53d238242aa9d35668376b2a285; no voters:
I20260924 04:49:20.327033 14884 raft_consensus.cc:2781] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:49:20.327240 14884 raft_consensus.cc:686] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 LEADER]: Becoming Leader. State: Replica: 94d7c53d238242aa9d35668376b2a285, State: Running, Role: LEADER
I20260924 04:49:20.327551 14884 consensus_queue.cc:237] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:20.330525 14688 catalog_manager.cc:5495] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94d7c53d238242aa9d35668376b2a285 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "94d7c53d238242aa9d35668376b2a285" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } health_report { overall_health: HEALTHY } } }
I20260924 04:49:20.363523 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.359802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.0.98.1:43849)
0924 04:49:20.360117 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:20.360124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:20.360135 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:49:20.360210 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:20.360212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:20.360268 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:20.360395 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:49:20.360662 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.361194 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.362025 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:20.362247 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:20.362331 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:20.362610 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:49:20.363124 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:49:20.363231 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:20.363300 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 04:49:20.363362 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
I20260924 04:49:20.367849 14970 raft_consensus.cc:1261] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354 [term 1 FOLLOWER]: Refusing update from remote peer 94d7c53d238242aa9d35668376b2a285: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:49:20.367849 15102 raft_consensus.cc:1261] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 1 FOLLOWER]: Refusing update from remote peer 94d7c53d238242aa9d35668376b2a285: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:49:20.368608 15169 consensus_queue.cc:1035] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:49:20.368804 14884 consensus_queue.cc:1035] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:49:20.378293 15173 mvcc.cc:205] Tried to move back new op lower bound from 7332763076060008448 to 7332763075904487424. Current Snapshot: MvccSnapshot[applied={T|T < 7332763076060008448}]
I20260924 04:49:20.384523 14688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:49:20.384677 14688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:49:20.386909 14688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Sending DeleteTablet for 3 replicas of tablet 5d91192d500a48ea990353f87044a58e
I20260924 04:49:20.387686 14950 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d91192d500a48ea990353f87044a58e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36580
I20260924 04:49:20.387830 14812 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d91192d500a48ea990353f87044a58e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54428
I20260924 04:49:20.387899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 14728
I20260924 04:49:20.388491 15082 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d91192d500a48ea990353f87044a58e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55622
I20260924 04:49:20.388933 15183 tablet_replica.cc:330] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1: stopping tablet replica
I20260924 04:49:20.389261 15183 raft_consensus.cc:2229] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:49:20.389592 15183 raft_consensus.cc:2258] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:49:20.391058 15183 ts_tablet_manager.cc:1905] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:49:20.391186 15182 tablet_replica.cc:330] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354: stopping tablet replica
I20260924 04:49:20.391470 15182 raft_consensus.cc:2229] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:49:20.391741 15182 raft_consensus.cc:2258] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:49:20.392998 15182 ts_tablet_manager.cc:1905] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:49:20.393781 15183 ts_tablet_manager.cc:1918] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:49:20.393894 15183 log.cc:1196] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/5d91192d500a48ea990353f87044a58e
I20260924 04:49:20.394217 15183 ts_tablet_manager.cc:1939] T 5d91192d500a48ea990353f87044a58e P 8b3e207b1de6487fbcb1016eecd4b2e1: Deleting consensus metadata
I20260924 04:49:20.395227 15182 ts_tablet_manager.cc:1918] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:49:20.395340 15182 log.cc:1196] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/5d91192d500a48ea990353f87044a58e
W20260924 04:49:20.395360 14677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43849 (error 108)
I20260924 04:49:20.395530 14676 catalog_manager.cc:4841] TS 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081): tablet 5d91192d500a48ea990353f87044a58e (table test-table [id=e88198644b1e4649adb4042e1a11cd12]) successfully deleted
I20260924 04:49:20.395638 15182 ts_tablet_manager.cc:1939] T 5d91192d500a48ea990353f87044a58e P 0a7f8b5c02a2405f85f86d7b3f377354: Deleting consensus metadata
I20260924 04:49:20.396333 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.396061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43849 (local address 127.0.0.1:54446)
0924 04:49:20.396187 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:20.396249 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 04:49:20.396494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 14882
W20260924 04:49:20.396479 14677 catalog_manager.cc:4584] TS 94d7c53d238242aa9d35668376b2a285 (127.0.98.1:43849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d91192d500a48ea990353f87044a58e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43849: connect: Connection refused (error 111)
I20260924 04:49:20.401757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 15019
I20260924 04:49:20.402781 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.402052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42131 (local address 127.0.0.1:36596)
0924 04:49:20.402628 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:20.402685 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495}
I20260924 04:49:20.406499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 14657
I20260924 04:49:20.411916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33651
--webserver_interface=127.0.98.62
--webserver_port=45239
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:20.522433 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:20.522722 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:20.522763 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:20.526985 15185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:49:20.527066 15185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:20.527093 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:20.527117 15185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:49:20.527137 15185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:49:20.532699 15185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:20.533835 15185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:20.534909 15185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:20.544034 15191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:20.544093 15185 server_base.cc:1034] running on GCE node
W20260924 04:49:20.544003 15190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:20.544001 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:20.545019 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:20.546321 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:20.547510 15185 hybrid_clock.cc:648] HybridClock initialized: now 1790225360547491 us; error 32 us; skew 500 ppm
Sep 24 04:49:20 dist-test-slave-hvcl krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:20.550463 15185 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:49:20.551424 15185 webserver.cc:460] Webserver started at http://127.0.98.62:45239/ using document root <none> and password file <none>
I20260924 04:49:20.551988 15185 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:20.552057 15185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:20.556095 15185 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:49:20.558519 15200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:20.559911 15185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:49:20.560045 15185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "64d60a82d7e74b81b1e2739c1140b868"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "082b86a05e9dc0a37573877508db8cb4"
server_key_iv: "7c9b4ab5bcb7c54780f61f558c15a02c"
server_key_version: "encryptionkey@0"
I20260924 04:49:20.560621 15185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:20.574425 15185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:20.577641 15185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:20.577824 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:20.584738 15185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33651
I20260924 04:49:20.585016 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33651 every 8 connection(s)
I20260924 04:49:20.585851 15185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:49:20.588238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 15185
I20260924 04:49:20.588639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43849
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33651
--webserver_port=45075
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:49:20.590503 15253 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:49:20.592705 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Bootstrap starting.
I20260924 04:49:20.615646 15253 log.cc:824] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:20.629912 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:49:20.630376 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: Bootstrap complete.
I20260924 04:49:20.635859 15253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } }
I20260924 04:49:20.636371 15253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d60a82d7e74b81b1e2739c1140b868, State: Initialized, Role: FOLLOWER
I20260924 04:49:20.636948 15253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } }
I20260924 04:49:20.637123 15253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:49:20.637202 15253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:49:20.637303 15253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:49:20.639894 15253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } }
I20260924 04:49:20.640265 15253 leader_election.cc:302] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64d60a82d7e74b81b1e2739c1140b868; no voters:
I20260924 04:49:20.640599 15253 leader_election.cc:288] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 04:49:20.640753 15257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:49:20.641023 15257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [term 3 LEADER]: Becoming Leader. State: Replica: 64d60a82d7e74b81b1e2739c1140b868, State: Running, Role: LEADER
I20260924 04:49:20.641352 15257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } }
I20260924 04:49:20.641888 15253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:49:20.642762 15258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "64d60a82d7e74b81b1e2739c1140b868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } } }
I20260924 04:49:20.642899 15258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:20.643093 15259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64d60a82d7e74b81b1e2739c1140b868. Latest consensus state: current_term: 3 leader_uuid: "64d60a82d7e74b81b1e2739c1140b868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d60a82d7e74b81b1e2739c1140b868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33651 } } }
I20260924 04:49:20.643190 15259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:20.647528 15272 catalog_manager.cc:1259] Loaded cluster ID: 6e2db1f69d4d43a48e50e048afe00f74
I20260924 04:49:20.647603 15272 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: loading cluster ID for follower catalog manager: success
W20260924 04:49:20.648710 15272 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 64d60a82d7e74b81b1e2739c1140b868: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260924 04:49:20.648809 15272 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260924 04:49:20.648883 15266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:49:20.651770 15266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:49:20.652277 15266 catalog_manager.cc:1259] Loaded cluster ID: 6e2db1f69d4d43a48e50e048afe00f74
I20260924 04:49:20.652336 15266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 04:49:20.652961 15266 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f90a3c269ad google::LogMessage::Fail() at ??:0
@ 0x7f90a3c2988a google::LogMessage::SendToLog() at ??:0
@ 0x7f90a3c264de google::LogMessage::Flush() at ??:0
@ 0x7f90a3c27a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f90ae565fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f90ae566d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f90ae563579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f90ae5a7640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f90a5e783d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f90a5e78c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f90a5e7a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f90a5e69512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f90a74ad6db start_thread at ??:0
@ 0x7f90a272461f clone at ??:0
W20260924 04:49:20.733093 15255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:20.733381 15255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:20.733444 15255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:20.737610 15255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:20.737675 15255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:20.737757 15255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:49:20.743213 15255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45075
--tserver_master_addrs=127.0.98.62:33651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:20.744349 15255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:20.745421 15255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:20.755021 15295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:20.755021 15294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:20.755303 15255 server_base.cc:1034] running on GCE node
W20260924 04:49:20.755645 15297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:20.756073 15255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:20.756704 15255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:20.757886 15255 hybrid_clock.cc:648] HybridClock initialized: now 1790225360757859 us; error 43 us; skew 500 ppm
Sep 24 04:49:20 dist-test-slave-hvcl krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:20.760807 15255 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:49:20.761726 15255 webserver.cc:460] Webserver started at http://127.0.98.1:45075/ using document root <none> and password file <none>
I20260924 04:49:20.762254 15255 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:20.762315 15255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:20.766256 15255 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:49:20.768507 15304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:20.769892 15255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:49:20.770009 15255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "94d7c53d238242aa9d35668376b2a285"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "89d3cd90b31f7b82f1257f74321d6789"
server_key_iv: "b720bc1c04fa7b4781846a3f59a76b3a"
server_key_version: "encryptionkey@0"
I20260924 04:49:20.770569 15255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:20.792125 15255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:20.795876 15255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:20.796075 15255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:20.796707 15255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:20.797878 15311 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:49:20.803610 15255 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:49:20.803691 15255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 04:49:20.803762 15255 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:49:20.805513 15255 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:49:20.805563 15255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:49:20.806054 15311 tablet_bootstrap.cc:492] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285: Bootstrap starting.
I20260924 04:49:20.815371 15255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43849
I20260924 04:49:20.815410 15418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43849 every 8 connection(s)
I20260924 04:49:20.816445 15255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:49:20.826390 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 15255
I20260924 04:49:20.826817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42131
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33651
--webserver_port=34305
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:49:20.827782 15311 log.cc:824] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:20.837364 15311 tablet_bootstrap.cc:492] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:49:20.837832 15311 tablet_bootstrap.cc:492] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285: Bootstrap complete.
I20260924 04:49:20.839210 15311 ts_tablet_manager.cc:1397] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.011s
I20260924 04:49:20.842701 15311 raft_consensus.cc:348] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:20.843297 15311 raft_consensus.cc:729] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d7c53d238242aa9d35668376b2a285, State: Initialized, Role: FOLLOWER
I20260924 04:49:20.843837 15311 consensus_queue.cc:260] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:20.844904 15311 ts_tablet_manager.cc:1428] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260924 04:49:20.882468 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:49:20.818376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:55119)
0924 04:49:20.818883 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:20.818936 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:20.819754 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:20.881674 (+ 61920us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":128,"threads_started":1}
W20260924 04:49:20.883002 15419 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33651 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33651: Client connection negotiation failed: client connection to 127.0.98.62:33651: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:49:20.883728 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.883471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:46859)
0924 04:49:20.883596 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:20.883667 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 04:49:20.884543 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:20.884313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:48273)
0924 04:49:20.884439 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:20.884488 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 04:49:20.884734 15419 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:49:20.945851 15422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:20.946141 15422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:20.946203 15422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:20.950419 15422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:20.950495 15422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:20.950577 15422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:49:20.956300 15422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34305
--tserver_master_addrs=127.0.98.62:33651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:20.957396 15422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:20.958429 15422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:20.968447 15430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:20.968523 15422 server_base.cc:1034] running on GCE node
W20260924 04:49:20.968472 15432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:20.968449 15429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:20.969148 15422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:20.969823 15422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:20.971030 15422 hybrid_clock.cc:648] HybridClock initialized: now 1790225360971005 us; error 34 us; skew 500 ppm
Sep 24 04:49:20 dist-test-slave-hvcl krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:20.973930 15422 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:49:20.974900 15422 webserver.cc:460] Webserver started at http://127.0.98.2:34305/ using document root <none> and password file <none>
I20260924 04:49:20.975509 15422 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:20.975580 15422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:20.979606 15422 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:49:20.981982 15439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:20.983562 15422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:49:20.983695 15422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "0a7f8b5c02a2405f85f86d7b3f377354"
format_stamp: "Formatted at 2026-09-24 04:49:18 on dist-test-slave-hvcl"
server_key: "14be1e6ef492798b993ad7acdd8853c1"
server_key_iv: "1cd495e17c2bbcca30b1c73dca5b903b"
server_key_version: "encryptionkey@0"
I20260924 04:49:20.984275 15422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:21.000836 15422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:21.004541 15422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:21.004761 15422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:21.005383 15422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:21.006217 15422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:21.006265 15422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:21.006307 15422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:21.006322 15422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:21.015125 15422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42131
I20260924 04:49:21.015180 15552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42131 every 8 connection(s)
I20260924 04:49:21.016155 15422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:49:21.019917 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:21.018324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:33679)
0924 04:49:21.019074 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:21.019189 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":105,"threads_started":1}
W20260924 04:49:21.020515 15553 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33651 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33651: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
I20260924 04:49:21.021083 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:21.020881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:58467)
0924 04:49:21.020978 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:21.021025 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 04:49:21.021641 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:21.021425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:56219)
0924 04:49:21.021539 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:21.021590 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 04:49:21.021795 15553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:49:21.024209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 15422
I20260924 04:49:21.024606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43081
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33651
--webserver_port=39853
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:21.135937 15556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:21.136210 15556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:21.136401 15556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:21.140638 15556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:21.140700 15556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:21.140767 15556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:49:21.146039 15556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39853
--tserver_master_addrs=127.0.98.62:33651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:21.147204 15556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:21.148241 15556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:21.157779 15564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:21.157779 15561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:21.157779 15562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:21.158540 15556 server_base.cc:1034] running on GCE node
I20260924 04:49:21.159001 15556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:21.159646 15556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:21.160805 15556 hybrid_clock.cc:648] HybridClock initialized: now 1790225361160786 us; error 31 us; skew 500 ppm
Sep 24 04:49:21 dist-test-slave-hvcl krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:21.163616 15556 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:49:21.164567 15556 webserver.cc:460] Webserver started at http://127.0.98.3:39853/ using document root <none> and password file <none>
I20260924 04:49:21.165112 15556 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:21.165176 15556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:21.169040 15556 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:49:21.171159 15571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:21.172447 15556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:49:21.172569 15556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "8b3e207b1de6487fbcb1016eecd4b2e1"
format_stamp: "Formatted at 2026-09-24 04:49:19 on dist-test-slave-hvcl"
server_key: "e5323fe3e209623459ede60510fde0a9"
server_key_iv: "e4303e0586d48e53ed7f6c3197038f04"
server_key_version: "encryptionkey@0"
I20260924 04:49:21.173166 15556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:21.198873 15556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:21.202314 15556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:21.202487 15556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:21.203092 15556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:21.203917 15556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:21.203964 15556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:21.204003 15556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:21.204023 15556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:21.212924 15556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43081
I20260924 04:49:21.212960 15684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43081 every 8 connection(s)
I20260924 04:49:21.214043 15556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:49:21.218015 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:21.216399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:32819)
0924 04:49:21.217118 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:21.217235 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":118,"threads_started":1}
W20260924 04:49:21.218611 15685 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33651 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33651: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
I20260924 04:49:21.219252 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:21.219031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:48491)
0924 04:49:21.219134 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:21.219185 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 04:49:21.219905 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:21.219683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:40519)
0924 04:49:21.219811 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:21.219854 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 04:49:21.220062 15685 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:49:21.221424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 15556
I20260924 04:49:21.885751 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:21.885302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:38353)
0924 04:49:21.885586 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:21.885638 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":70,"threads_started":1}
I20260924 04:49:22.022833 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:22.022413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:58387)
0924 04:49:22.022659 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:22.022721 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260924 04:49:22.221195 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:22.220718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:40101)
0924 04:49:22.221002 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:22.221073 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":75,"threads_started":1}
I20260924 04:49:22.639580 15692 raft_consensus.cc:482] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:49:22.639775 15692 raft_consensus.cc:504] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:22.640892 15692 leader_election.cc:288] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131), 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081)
Sep 24 04:49:22 dist-test-slave-hvcl krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 04:49:22 dist-test-slave-hvcl krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 04:49:22.657248 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:22.641189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43081 (local address 127.0.98.1:38727)
0924 04:49:22.641953 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:22.641963 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:22.642049 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:22.643638 (+ 1589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:22.643645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:22.644226 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:22.645001 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:22.645024 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:22.646994 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:22.646997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:22.648589 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:22.648600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:22.649624 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:22.649628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:22.649939 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:22.650624 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:22.650647 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:22.652771 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:22.654763 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:22.654766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:22.654768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:22.655105 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:22.655443 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:22.655446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:22.655448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:22.655588 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:22.656225 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:22.656234 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:22.656484 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 04:49:22.656647 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 04:49:22.657029 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":784,"spinlock_wait_cycles":4736,"thread_start_us":77,"threads_started":1}
I20260924 04:49:22.657255 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:22.641398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50065 (local address 127.0.98.2:42131)
0924 04:49:22.641716 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:22.641732 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:22.641781 (+ 49us) server_negotiation.cc:407] Connection header received
0924 04:49:22.642644 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:22.642653 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:22.643167 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:22.643510 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:22.645213 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:22.646787 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:22.648723 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:22.649473 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:22.649642 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:22.652741 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:22.652746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:22.652760 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:22.652792 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:22.654582 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:22.655215 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:22.655221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:22.655227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:22.655355 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:22.655698 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:22.655702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:22.655711 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:22.656107 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:22.656225 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:22.656813 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 04:49:22.657068 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20260924 04:49:22.657255 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:22.641346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38727 (local address 127.0.98.3:43081)
0924 04:49:22.641715 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:22.641740 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:22.642059 (+ 319us) server_negotiation.cc:407] Connection header received
0924 04:49:22.642829 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:22.642835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:22.643166 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:22.643510 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:22.645213 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:22.646787 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:22.648723 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:22.649472 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:22.649642 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:22.652981 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:22.652987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:22.653003 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:22.653038 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:22.654660 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:22.655215 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:22.655220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:22.655224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:22.655348 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:22.655690 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:22.655692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:22.655693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:22.656108 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:22.656224 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:22.656809 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 04:49:22.657067 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":57,"threads_started":1}
I20260924 04:49:22.657270 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:22.641189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42131 (local address 127.0.98.1:50065)
0924 04:49:22.641546 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:22.641561 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:22.641672 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:22.643638 (+ 1966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:22.643645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:22.644226 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:22.645001 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:22.645024 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:22.646965 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:22.646993 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:22.648588 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:22.648601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:22.649624 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:22.649628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:22.649943 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:22.650633 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:22.650652 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:22.652499 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:22.654710 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:22.654716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:22.654731 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:22.655101 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:22.655453 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:22.655457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:22.655459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:22.655594 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:22.656223 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:22.656229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:22.656486 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 04:49:22.656646 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 04:49:22.657104 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":549,"spinlock_wait_cycles":18944,"thread_start_us":84,"threads_started":1}
W20260924 04:49:22.660125 15307 leader_election.cc:341] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081): Not found: Tablet not found: 5d91192d500a48ea990353f87044a58e
W20260924 04:49:22.660303 15305 leader_election.cc:341] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131): Not found: Tablet not found: 5d91192d500a48ea990353f87044a58e
I20260924 04:49:22.660395 15305 leader_election.cc:302] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94d7c53d238242aa9d35668376b2a285; no voters: 0a7f8b5c02a2405f85f86d7b3f377354, 8b3e207b1de6487fbcb1016eecd4b2e1
I20260924 04:49:22.660647 15692 raft_consensus.cc:2726] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:49:22.886931 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:22.886612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:57769)
0924 04:49:22.886772 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:22.886843 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 04:49:23.024062 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:23.023603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:52813)
0924 04:49:23.023889 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:23.023944 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
I20260924 04:49:23.222524 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:23.222029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:50391)
0924 04:49:23.222359 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:23.222425 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
I20260924 04:49:23.888248 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:23.887834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:55123)
0924 04:49:23.888085 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:23.888147 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260924 04:49:24.025394 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:24.024939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:35617)
0924 04:49:24.025229 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:24.025297 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
I20260924 04:49:24.223839 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:24.223364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:37727)
0924 04:49:24.223655 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:24.223721 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
I20260924 04:49:24.478858 15702 raft_consensus.cc:482] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:49:24.479022 15702 raft_consensus.cc:504] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:24.479447 15702 leader_election.cc:288] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131), 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081)
W20260924 04:49:24.480235 15307 leader_election.cc:341] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081): Not found: Tablet not found: 5d91192d500a48ea990353f87044a58e
W20260924 04:49:24.480327 15305 leader_election.cc:341] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131): Not found: Tablet not found: 5d91192d500a48ea990353f87044a58e
I20260924 04:49:24.480422 15305 leader_election.cc:302] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94d7c53d238242aa9d35668376b2a285; no voters: 0a7f8b5c02a2405f85f86d7b3f377354, 8b3e207b1de6487fbcb1016eecd4b2e1
I20260924 04:49:24.480577 15702 raft_consensus.cc:2726] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:49:24.889557 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:24.889088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:39615)
0924 04:49:24.889383 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:24.889446 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
I20260924 04:49:25.026635 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:25.026170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:48685)
0924 04:49:25.026450 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:25.026515 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
I20260924 04:49:25.225183 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:25.224670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:46489)
0924 04:49:25.225012 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:25.225077 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":81,"threads_started":1}
I20260924 04:49:25.890942 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:25.890467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:38113)
0924 04:49:25.890771 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:25.890843 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":74,"threads_started":1}
I20260924 04:49:26.028093 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:26.027549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:52515)
0924 04:49:26.027838 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:26.027971 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260924 04:49:26.226555 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:26.226081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:35815)
0924 04:49:26.226409 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:26.226461 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":157,"threads_started":1}
I20260924 04:49:26.892096 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:26.891699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:42263)
0924 04:49:26.891940 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:26.891999 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":70,"threads_started":1}
I20260924 04:49:27.029440 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:27.028898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:47281)
0924 04:49:27.029209 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:27.029336 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":89,"threads_started":1}
I20260924 04:49:27.182721 15711 raft_consensus.cc:482] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:49:27.182861 15711 raft_consensus.cc:504] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:27.183296 15711 leader_election.cc:288] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131), 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081)
W20260924 04:49:27.184196 15305 leader_election.cc:341] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131): Not found: Tablet not found: 5d91192d500a48ea990353f87044a58e
W20260924 04:49:27.184326 15307 leader_election.cc:341] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081): Not found: Tablet not found: 5d91192d500a48ea990353f87044a58e
I20260924 04:49:27.184433 15307 leader_election.cc:302] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94d7c53d238242aa9d35668376b2a285; no voters: 0a7f8b5c02a2405f85f86d7b3f377354, 8b3e207b1de6487fbcb1016eecd4b2e1
I20260924 04:49:27.184610 15711 raft_consensus.cc:2726] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:49:27.227872 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:27.227394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:52211)
0924 04:49:27.227694 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:27.227760 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260924 04:49:27.893409 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:27.892963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:48423)
0924 04:49:27.893220 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:27.893278 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
I20260924 04:49:28.030807 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:28.030259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:54537)
0924 04:49:28.030625 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:28.030691 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":95,"threads_started":1}
I20260924 04:49:28.229261 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:28.228735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:46171)
0924 04:49:28.229094 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:28.229162 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260924 04:49:28.894553 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:28.894133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:43017)
0924 04:49:28.894391 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:28.894453 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260924 04:49:29.032266 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:29.031793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:40727)
0924 04:49:29.032084 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:29.032165 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
I20260924 04:49:29.230613 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:29.230118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:45789)
0924 04:49:29.230449 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:29.230514 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":123,"threads_started":1}
I20260924 04:49:29.895757 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:29.895349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:59157)
0924 04:49:29.895607 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:29.895664 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":92,"threads_started":1}
I20260924 04:49:30.033601 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:30.033145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:34397)
0924 04:49:30.033434 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:30.033500 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260924 04:49:30.232002 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:30.231528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:53369)
0924 04:49:30.231816 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:30.231893 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":90,"threads_started":1}
I20260924 04:49:30.426774 15722 raft_consensus.cc:482] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:49:30.426914 15722 raft_consensus.cc:504] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:30.427390 15722 leader_election.cc:288] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131), 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081)
W20260924 04:49:30.428203 15307 leader_election.cc:341] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081): Not found: Tablet not found: 5d91192d500a48ea990353f87044a58e
W20260924 04:49:30.428298 15305 leader_election.cc:341] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131): Not found: Tablet not found: 5d91192d500a48ea990353f87044a58e
I20260924 04:49:30.428402 15305 leader_election.cc:302] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94d7c53d238242aa9d35668376b2a285; no voters: 0a7f8b5c02a2405f85f86d7b3f377354, 8b3e207b1de6487fbcb1016eecd4b2e1
I20260924 04:49:30.428570 15722 raft_consensus.cc:2726] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:49:30.897073 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:30.896615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:56365)
0924 04:49:30.896912 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:30.896972 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":90,"threads_started":1}
I20260924 04:49:31.035053 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:31.034540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:38385)
0924 04:49:31.034844 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:31.034910 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260924 04:49:31.233397 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:31.232920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:45787)
0924 04:49:31.233203 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:31.233298 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260924 04:49:31.898160 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:31.897771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:38461)
0924 04:49:31.898007 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:31.898060 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260924 04:49:32.036406 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:32.035913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:53639)
0924 04:49:32.036224 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:32.036300 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":91,"threads_started":1}
I20260924 04:49:32.234752 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:32.234292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:46129)
0924 04:49:32.234585 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:32.234654 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":94,"threads_started":1}
I20260924 04:49:32.899467 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:32.899004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:57237)
0924 04:49:32.899297 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:32.899365 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
I20260924 04:49:33.037668 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:33.037241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:60491)
0924 04:49:33.037505 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:33.037566 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260924 04:49:33.236017 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:33.235572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:39133)
0924 04:49:33.235854 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:33.235917 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":93,"threads_started":1}
I20260924 04:49:33.900727 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:33.900263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:53825)
0924 04:49:33.900556 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:33.900621 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":73,"threads_started":1}
I20260924 04:49:34.038884 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:34.038419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:60001)
0924 04:49:34.038724 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:34.038786 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":85,"threads_started":1}
I20260924 04:49:34.237390 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:34.236841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:37563)
0924 04:49:34.237207 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:34.237279 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":97,"threads_started":1}
I20260924 04:49:34.873920 15735 raft_consensus.cc:482] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:49:34.874073 15735 raft_consensus.cc:504] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7f8b5c02a2405f85f86d7b3f377354" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42131 } } peers { permanent_uuid: "8b3e207b1de6487fbcb1016eecd4b2e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43081 } } peers { permanent_uuid: "94d7c53d238242aa9d35668376b2a285" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43849 } }
I20260924 04:49:34.874521 15735 leader_election.cc:288] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131), 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081)
W20260924 04:49:34.875252 15305 leader_election.cc:341] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a7f8b5c02a2405f85f86d7b3f377354 (127.0.98.2:42131): Not found: Tablet not found: 5d91192d500a48ea990353f87044a58e
W20260924 04:49:34.875362 15307 leader_election.cc:341] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b3e207b1de6487fbcb1016eecd4b2e1 (127.0.98.3:43081): Not found: Tablet not found: 5d91192d500a48ea990353f87044a58e
I20260924 04:49:34.875440 15307 leader_election.cc:302] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94d7c53d238242aa9d35668376b2a285; no voters: 0a7f8b5c02a2405f85f86d7b3f377354, 8b3e207b1de6487fbcb1016eecd4b2e1
I20260924 04:49:34.875562 15735 raft_consensus.cc:2726] T 5d91192d500a48ea990353f87044a58e P 94d7c53d238242aa9d35668376b2a285 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:49:34.901973 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:34.901516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:45367)
0924 04:49:34.901809 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:34.901874 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
I20260924 04:49:35.039969 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:35.039538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:52177)
0924 04:49:35.039809 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:35.039870 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
I20260924 04:49:35.238539 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:35.238116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.3:49163)
0924 04:49:35.238384 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:35.238444 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
I20260924 04:49:35.903431 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:35.903021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.1:48455)
0924 04:49:35.903274 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:35.903335 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260924 04:49:36.041525 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:36.041061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33651 (local address 127.0.98.2:60379)
0924 04:49:36.041363 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:36.041426 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":103,"threads_started":1}
I20260924 04:49:36.229311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 15255
I20260924 04:49:36.234303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 15422
I20260924 04:49:36.239177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 15556
2026-09-24T04:49:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19133 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:49:36 dist-test-slave-hvcl krb5kdc[15743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:49:36 dist-test-slave-hvcl krb5kdc[15743](info): set up 2 sockets
Sep 24 04:49:36 dist-test-slave-hvcl krb5kdc[15743](info): commencing operation
krb5kdc: starting...
W20260924 04:49:37.203992 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:49:37 dist-test-slave-hvcl krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:49:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:49:37.326350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34241
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:37.436507 15756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:37.436815 15756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:37.436856 15756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:37.440889 15756 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:49:37.440955 15756 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:49:37.440970 15756 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:49:37.440982 15756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:49:37.441000 15756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:37.441016 15756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:37.441033 15756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:49:37.441049 15756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:49:37.446484 15756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:37.447589 15756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:37.448581 15756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:37.457691 15762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:37.457693 15761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:37.457693 15764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:37.457758 15756 server_base.cc:1034] running on GCE node
I20260924 04:49:37.458549 15756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:37.459779 15756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:37.460960 15756 hybrid_clock.cc:648] HybridClock initialized: now 1790225377460930 us; error 43 us; skew 500 ppm
Sep 24 04:49:37 dist-test-slave-hvcl krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:37.463842 15756 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:49:37.464778 15756 webserver.cc:460] Webserver started at http://127.0.98.62:42435/ using document root <none> and password file <none>
I20260924 04:49:37.465324 15756 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:37.465386 15756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:37.465591 15756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:49:37.467739 15756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "ebcdcf67db39499293d8a608a5cb4a79"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "ade310c1610f57b6549f70ad1bff6cd3"
server_key_iv: "00ddd90136f7b50cb3732425f1320504"
server_key_version: "encryptionkey@0"
I20260924 04:49:37.468307 15756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "ebcdcf67db39499293d8a608a5cb4a79"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "ade310c1610f57b6549f70ad1bff6cd3"
server_key_iv: "00ddd90136f7b50cb3732425f1320504"
server_key_version: "encryptionkey@0"
I20260924 04:49:37.472324 15756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 04:49:37.475041 15771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:37.476408 15756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:49:37.476541 15756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "ebcdcf67db39499293d8a608a5cb4a79"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "ade310c1610f57b6549f70ad1bff6cd3"
server_key_iv: "00ddd90136f7b50cb3732425f1320504"
server_key_version: "encryptionkey@0"
I20260924 04:49:37.476662 15756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:37.516727 15756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:37.519950 15756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:37.520145 15756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:37.527084 15756 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34241
I20260924 04:49:37.527390 15823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34241 every 8 connection(s)
I20260924 04:49:37.528079 15756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:49:37.531131 15824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:37.533034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 15756
I20260924 04:49:37.533166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:49:37.533458 392 external_mini_cluster.cc:1428] Setting key 87c93aeb4b257d9c7eb55a8731d546f9
I20260924 04:49:37.538197 15824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Bootstrap starting.
I20260924 04:49:37.540750 15824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:37.541479 15824 log.cc:824] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:37.543488 15824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: No bootstrap required, opened a new log
Sep 24 04:49:37 dist-test-slave-hvcl krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:37.547017 15824 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 04:49:37.547277 15824 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:37.547338 15824 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcdcf67db39499293d8a608a5cb4a79, State: Initialized, Role: FOLLOWER
I20260924 04:49:37.547814 15824 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 04:49:37.547984 15824 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:49:37.548059 15824 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:49:37.548161 15824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:37.549294 15824 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 04:49:37.549676 15824 leader_election.cc:302] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebcdcf67db39499293d8a608a5cb4a79; no voters:
I20260924 04:49:37.550037 15824 leader_election.cc:288] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:49:37.550140 15829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:49:37.550359 15829 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 1 LEADER]: Becoming Leader. State: Replica: ebcdcf67db39499293d8a608a5cb4a79, State: Running, Role: LEADER
I20260924 04:49:37.550454 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:37.536314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52220 (local address 127.0.98.62:34241)
0924 04:49:37.536774 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:37.536793 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:37.536834 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:49:37.537705 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:37.537733 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:37.538217 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:37.538661 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:37.539617 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:37.540606 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:37.541518 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:37.542062 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:37.542208 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:37.544899 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:37.544904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:37.544917 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:37.544945 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:37.547106 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:37.547580 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:37.547586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:37.547592 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:37.547723 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:37.548045 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:37.548049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:37.548050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:37.548569 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:37.548765 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:37.549039 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 04:49:37.549359 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":154,"threads_started":1}
I20260924 04:49:37.550702 15829 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 04:49:37.551340 15824 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:49:37.552306 15831 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebcdcf67db39499293d8a608a5cb4a79. Latest consensus state: current_term: 1 leader_uuid: "ebcdcf67db39499293d8a608a5cb4a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } } }
I20260924 04:49:37.552431 15831 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:37.552752 15830 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebcdcf67db39499293d8a608a5cb4a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } } }
I20260924 04:49:37.552855 15830 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:37.554407 15833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:49:37.557904 15833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:49:37.564261 15833 catalog_manager.cc:1347] Generated new cluster ID: f32e0c94c01e4ae49615adfb3984a307
I20260924 04:49:37.564345 15833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:49:37.576424 15833 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:49:37.577761 15833 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:49:37.592840 15833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Generated new TSK 0
I20260924 04:49:37.593637 15833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:49:37.631489 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:37.744303 15850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:37.744630 15850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:37.744678 15850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:37.748939 15850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:37.749018 15850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:37.749078 15850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:49:37.754624 15850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:37.755837 15850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:37.756903 15850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:37.766595 15856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:37.766597 15858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:37.766597 15855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:37.767027 15850 server_base.cc:1034] running on GCE node
I20260924 04:49:37.767458 15850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:37.768105 15850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:37.769294 15850 hybrid_clock.cc:648] HybridClock initialized: now 1790225377769271 us; error 33 us; skew 500 ppm
Sep 24 04:49:37 dist-test-slave-hvcl krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:37.772148 15850 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:49:37.773309 15850 webserver.cc:460] Webserver started at http://127.0.98.1:36721/ using document root <none> and password file <none>
I20260924 04:49:37.773833 15850 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:37.773890 15850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:37.774058 15850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:49:37.776129 15850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "9b3070d2261747659b8c90867bd9ae36"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "043bca0ce576c43476cdab4a3ce52d37"
server_key_iv: "809d7e5fbaa6be9b4446099daceecaca"
server_key_version: "encryptionkey@0"
I20260924 04:49:37.776638 15850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9b3070d2261747659b8c90867bd9ae36"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "043bca0ce576c43476cdab4a3ce52d37"
server_key_iv: "809d7e5fbaa6be9b4446099daceecaca"
server_key_version: "encryptionkey@0"
I20260924 04:49:37.780476 15850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:49:37.783214 15865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:37.784510 15850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:49:37.784631 15850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "9b3070d2261747659b8c90867bd9ae36"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "043bca0ce576c43476cdab4a3ce52d37"
server_key_iv: "809d7e5fbaa6be9b4446099daceecaca"
server_key_version: "encryptionkey@0"
I20260924 04:49:37.784749 15850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:37.798897 15850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:37.802184 15850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:37.802356 15850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:37.802991 15850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:37.803841 15850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:37.803889 15850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:37.803930 15850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:37.803958 15850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:37.812644 15850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42927
I20260924 04:49:37.812670 15978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42927 every 8 connection(s)
I20260924 04:49:37.813575 15850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:49:37.817860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 15850
I20260924 04:49:37.818007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:49:37.818311 392 external_mini_cluster.cc:1428] Setting key 2e11e026cf5cee1e5ce7816016cf071d
Sep 24 04:49:37 dist-test-slave-hvcl krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:49:37.828531 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:37.815544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40195 (local address 127.0.98.62:34241)
0924 04:49:37.815680 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:37.815684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:37.816800 (+ 1116us) server_negotiation.cc:407] Connection header received
0924 04:49:37.817750 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:37.817753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:37.817803 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:37.817886 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:37.819458 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:37.820073 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:37.821795 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:37.822033 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:37.822120 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:37.824957 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:37.824961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:37.824964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:37.824993 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:37.826378 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:37.827029 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:37.827031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:37.827032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:37.827111 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:37.827486 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:37.827488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:37.827489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:37.827681 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:37.827763 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:37.828268 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 04:49:37.828401 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:49:37.829370 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:37.815950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:40195)
0924 04:49:37.816643 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:37.816687 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:37.817472 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:37.818103 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:37.818113 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:37.818633 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:37.819296 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:37.819316 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:37.820262 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:37.820265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:37.821644 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:37.821658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:37.822206 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:37.822219 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:37.822425 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:37.823075 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:37.823092 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:37.824733 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:37.826505 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:37.826512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:37.826530 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:37.826895 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:37.827203 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:37.827206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:37.827208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:37.827390 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:37.827776 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:37.827781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:37.828037 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 04:49:37.828196 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 04:49:37.828486 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:49:37.830597 15979 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34241
I20260924 04:49:37.830873 15979 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:37.831513 15979 heartbeater.cc:511] Master 127.0.98.62:34241 requested a full tablet report, sending...
I20260924 04:49:37.833269 15788 ts_manager.cc:194] Registered new tserver with Master: 9b3070d2261747659b8c90867bd9ae36 (127.0.98.1:42927)
I20260924 04:49:37.834852 15788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:49:37.848129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:37.963194 15984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:37.963544 15984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:37.963613 15984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:37.968127 15984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:37.968215 15984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:37.968303 15984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:49:37.974743 15984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:37.975903 15984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:37.976902 15984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:37.986871 15992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:37.986852 15990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:37.987063 15984 server_base.cc:1034] running on GCE node
W20260924 04:49:37.986874 15989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:37.987778 15984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:37.988528 15984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:37.989705 15984 hybrid_clock.cc:648] HybridClock initialized: now 1790225377989676 us; error 45 us; skew 500 ppm
Sep 24 04:49:37 dist-test-slave-hvcl krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:37.992573 15984 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:49:37.993487 15984 webserver.cc:460] Webserver started at http://127.0.98.2:41753/ using document root <none> and password file <none>
I20260924 04:49:37.994058 15984 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:37.994124 15984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:37.994328 15984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:49:37.996496 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "ccc980eb97cacd6345412a992b1b028f"
server_key_iv: "eda93f12a8b122c18bc8365cb12a4118"
server_key_version: "encryptionkey@0"
I20260924 04:49:37.997081 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "ccc980eb97cacd6345412a992b1b028f"
server_key_iv: "eda93f12a8b122c18bc8365cb12a4118"
server_key_version: "encryptionkey@0"
I20260924 04:49:38.001374 15984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:49:38.004387 15999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:38.005963 15984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:49:38.006121 15984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "ccc980eb97cacd6345412a992b1b028f"
server_key_iv: "eda93f12a8b122c18bc8365cb12a4118"
server_key_version: "encryptionkey@0"
I20260924 04:49:38.006255 15984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:38.030779 15984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:38.034087 15984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:38.034284 15984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:38.034902 15984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:38.035778 15984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:38.035826 15984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:38.035880 15984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:38.035910 15984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:38.044768 15984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42657
I20260924 04:49:38.044790 16112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42657 every 8 connection(s)
I20260924 04:49:38.045749 15984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:49:38.055007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 15984
I20260924 04:49:38.055183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:49:38.055470 392 external_mini_cluster.cc:1428] Setting key e6e3aac1bde0e7496f6b00b3013128a5
Sep 24 04:49:38 dist-test-slave-hvcl krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:38.060334 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.047730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34991 (local address 127.0.98.62:34241)
0924 04:49:38.047874 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:38.047878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:38.048986 (+ 1108us) server_negotiation.cc:407] Connection header received
0924 04:49:38.049872 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:38.049875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:38.049923 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:38.049998 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:38.051227 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.051794 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.053404 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.053654 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.053738 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.056501 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:38.056504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:38.056506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:38.056531 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:38.058018 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:38.058660 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:38.058663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:38.058664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:38.058751 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:38.059144 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:38.059148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:38.059149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:38.059348 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:38.059435 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:38.059999 (+ 564us) server_negotiation.cc:299] Negotiation successful
0924 04:49:38.060162 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:49:38.061196 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.048141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:34991)
0924 04:49:38.048831 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:38.048874 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:38.049679 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:38.050163 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:38.050170 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:38.050523 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:38.051026 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:38.051041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.051919 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.051922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:38.053208 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:38.053218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.053815 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.053818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:38.053970 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.054634 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:38.054651 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:38.056262 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:38.058141 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:38.058148 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:38.058168 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:38.058530 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:38.058842 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:38.058845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:38.058847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:38.059056 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:38.059450 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:38.059457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:38.059740 (+ 283us) client_negotiation.cc:769] Sending connection context
0924 04:49:38.059914 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 04:49:38.060290 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":128,"threads_started":1}
I20260924 04:49:38.062507 16113 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34241
I20260924 04:49:38.062784 16113 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:38.063338 16113 heartbeater.cc:511] Master 127.0.98.62:34241 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:49:38.064487 15788 ts_manager.cc:194] Registered new tserver with Master: 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657)
I20260924 04:49:38.065060 15788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:49:38.082484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:38.197477 16118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:38.197772 16118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:38.197834 16118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:38.202014 16118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:38.202098 16118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:38.202183 16118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:49:38.207999 16118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:38.209154 16118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:38.210232 16118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:38.220099 16123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:38.220156 16126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:38.220413 16118 server_base.cc:1034] running on GCE node
W20260924 04:49:38.220156 16124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:38.220829 16118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:38.221517 16118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:38.222685 16118 hybrid_clock.cc:648] HybridClock initialized: now 1790225378222667 us; error 31 us; skew 500 ppm
Sep 24 04:49:38 dist-test-slave-hvcl krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:38.225502 16118 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:49:38.226385 16118 webserver.cc:460] Webserver started at http://127.0.98.3:45287/ using document root <none> and password file <none>
I20260924 04:49:38.226939 16118 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:38.227038 16118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:38.227274 16118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:49:38.229390 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "e20568f2f5e74956bc62367868e4c8a4"
format_stamp: "Formatted at 2026-09-24 04:49:38 on dist-test-slave-hvcl"
server_key: "a2e28863cc31bb4ebb9f92e78de14743"
server_key_iv: "61ed705fcfaf6e88a217df84889830cc"
server_key_version: "encryptionkey@0"
I20260924 04:49:38.229934 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e20568f2f5e74956bc62367868e4c8a4"
format_stamp: "Formatted at 2026-09-24 04:49:38 on dist-test-slave-hvcl"
server_key: "a2e28863cc31bb4ebb9f92e78de14743"
server_key_iv: "61ed705fcfaf6e88a217df84889830cc"
server_key_version: "encryptionkey@0"
I20260924 04:49:38.233771 16118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 04:49:38.236387 16133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:38.237746 16118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:49:38.237877 16118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "e20568f2f5e74956bc62367868e4c8a4"
format_stamp: "Formatted at 2026-09-24 04:49:38 on dist-test-slave-hvcl"
server_key: "a2e28863cc31bb4ebb9f92e78de14743"
server_key_iv: "61ed705fcfaf6e88a217df84889830cc"
server_key_version: "encryptionkey@0"
I20260924 04:49:38.237995 16118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:38.268569 16118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:38.272284 16118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:38.272500 16118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:38.273144 16118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:38.274016 16118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:38.274067 16118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:38.274121 16118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:38.274148 16118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:38.283326 16118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42755
I20260924 04:49:38.283330 16246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42755 every 8 connection(s)
I20260924 04:49:38.284315 16118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:49:38.289325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 16118
I20260924 04:49:38.289446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:49:38.289709 392 external_mini_cluster.cc:1428] Setting key 88c8a249e61b916491b5b8cda7cb6d69
Sep 24 04:49:38 dist-test-slave-hvcl krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:38.297820 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.286206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60867 (local address 127.0.98.62:34241)
0924 04:49:38.286358 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:38.286361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:38.287475 (+ 1114us) server_negotiation.cc:407] Connection header received
0924 04:49:38.288473 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:38.288476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:38.288532 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:38.288609 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:38.290022 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.290566 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.291780 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.291996 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.292093 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.294516 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:38.294519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:38.294521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:38.294546 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:38.295896 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:38.296377 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:38.296380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:38.296382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:38.296456 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:38.296769 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:38.296772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:38.296773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:38.296954 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:38.297037 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:38.297531 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 04:49:38.297653 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:49:38.298648 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.286564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:60867)
0924 04:49:38.287309 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:38.287355 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:38.288203 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:38.288809 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:38.288817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:38.289240 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:38.289857 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:38.289874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.290704 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.290707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:38.291648 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:38.291665 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.292152 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.292156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:38.292330 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.292906 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:38.292924 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:38.294356 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:38.296009 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:38.296015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:38.296032 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:38.296287 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:38.296534 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:38.296537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:38.296539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:38.296686 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:38.297047 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:38.297055 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:38.297311 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 04:49:38.297479 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 04:49:38.297828 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":121,"threads_started":1}
I20260924 04:49:38.299757 16247 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34241
I20260924 04:49:38.299968 16247 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:38.300441 16247 heartbeater.cc:511] Master 127.0.98.62:34241 requested a full tablet report, sending...
I20260924 04:49:38.301401 15788 ts_manager.cc:194] Registered new tserver with Master: e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755)
I20260924 04:49:38.302019 15788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60867
I20260924 04:49:38.303643 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:49:38.313054 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.305642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.0.98.62:34241)
0924 04:49:38.305785 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:38.305790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:38.305932 (+ 142us) server_negotiation.cc:407] Connection header received
0924 04:49:38.306011 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:38.306020 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:38.306064 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:38.306128 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:38.306781 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.307374 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.308248 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.308457 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.308533 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.309930 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:38.309933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:38.309935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:38.309961 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:38.311471 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:38.311887 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:38.311890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:38.311891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:38.311972 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:38.312251 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:38.312254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:38.312255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:38.312487 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:38.312600 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:38.312801 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 04:49:38.312901 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:49:38.316183 15788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:49:38.319339 15788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:49:38.334504 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.328395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.0.98.2:42657)
0924 04:49:38.328943 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:38.328956 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:38.329265 (+ 309us) server_negotiation.cc:407] Connection header received
0924 04:49:38.329477 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:38.329484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:38.329708 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:38.329839 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:38.330356 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.331548 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.333212 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.333800 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.333901 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.334062 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:38.334214 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 04:49:38.334334 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":80,"threads_started":1}
I20260924 04:49:38.334492 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.328404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42755 (local address 127.0.0.1:52326)
0924 04:49:38.329112 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:38.329166 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:38.329299 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:38.330313 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:38.330316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:38.330334 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:38.330411 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:38.330417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.331988 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.331991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:38.332968 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:38.332977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.334033 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.334036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:38.334134 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.334155 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:49:38.334220 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:49:38.334343 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":29,"thread_start_us":117,"threads_started":1}
I20260924 04:49:38.334852 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.328880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.0.98.3:42755)
0924 04:49:38.329866 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:38.329878 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:38.329899 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:49:38.329968 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:38.329973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:38.330162 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:38.330541 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:38.330731 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.331833 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.333087 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.333893 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.334302 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.334434 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:38.334482 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 04:49:38.334715 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":195,"threads_started":1}
I20260924 04:49:38.334945 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.328274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42657 (local address 127.0.0.1:37038)
0924 04:49:38.329112 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:38.329166 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:38.329323 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:38.330057 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:38.330063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:38.330085 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:38.330232 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:38.330240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.331683 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.331687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:38.333121 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:38.333129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.333919 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.333923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:38.334029 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.334064 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:49:38.334174 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 04:49:38.334850 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":40,"thread_start_us":104,"threads_started":1}
I20260924 04:49:38.335083 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.329051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42927 (local address 127.0.0.1:56894)
0924 04:49:38.329527 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:38.329538 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:38.329610 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:38.330252 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:38.330255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:38.330271 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:38.330363 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:38.330369 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.331421 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.331424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:38.333030 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:38.333041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.334199 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.334202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:38.334293 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.334312 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:49:38.334370 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 04:49:38.334440 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":78,"threads_started":1}
I20260924 04:49:38.335301 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.329463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.0.98.1:42927)
0924 04:49:38.329884 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:38.329898 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:38.329919 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:49:38.329984 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:38.329988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:38.330128 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:38.330256 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:38.330453 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.331325 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.333147 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.333910 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.334001 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.334143 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:38.334554 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 04:49:38.334865 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":110,"threads_started":1}
I20260924 04:49:38.337884 16181 tablet_service.cc:1467] Processing CreateTablet for tablet c78fe5d52fca44dd8b4d715a183a8d32 (DEFAULT_TABLE table=test-table [id=9e9d2cb4feab46fa8e9fe00c29a79768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:38.337934 15913 tablet_service.cc:1467] Processing CreateTablet for tablet c78fe5d52fca44dd8b4d715a183a8d32 (DEFAULT_TABLE table=test-table [id=9e9d2cb4feab46fa8e9fe00c29a79768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:38.337936 16047 tablet_service.cc:1467] Processing CreateTablet for tablet c78fe5d52fca44dd8b4d715a183a8d32 (DEFAULT_TABLE table=test-table [id=9e9d2cb4feab46fa8e9fe00c29a79768]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:38.338855 16181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78fe5d52fca44dd8b4d715a183a8d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:38.338940 15913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78fe5d52fca44dd8b4d715a183a8d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:38.338940 16047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78fe5d52fca44dd8b4d715a183a8d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:38.345369 16264 tablet_bootstrap.cc:492] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4: Bootstrap starting.
I20260924 04:49:38.346802 16266 tablet_bootstrap.cc:492] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db: Bootstrap starting.
I20260924 04:49:38.346829 16265 tablet_bootstrap.cc:492] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: Bootstrap starting.
I20260924 04:49:38.347774 16264 tablet_bootstrap.cc:654] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:38.348881 16264 log.cc:824] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:38.349109 16265 tablet_bootstrap.cc:654] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:38.349448 16266 tablet_bootstrap.cc:654] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:38.350014 16265 log.cc:824] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:38.350224 16266 log.cc:824] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:38.352101 16265 tablet_bootstrap.cc:492] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: No bootstrap required, opened a new log
I20260924 04:49:38.352162 16264 tablet_bootstrap.cc:492] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4: No bootstrap required, opened a new log
I20260924 04:49:38.352157 16266 tablet_bootstrap.cc:492] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db: No bootstrap required, opened a new log
I20260924 04:49:38.352308 16265 ts_tablet_manager.cc:1397] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:49:38.352346 16264 ts_tablet_manager.cc:1397] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 04:49:38.352424 16266 ts_tablet_manager.cc:1397] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:49:38.356180 16265 raft_consensus.cc:348] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:38.356441 16265 raft_consensus.cc:374] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:38.356493 16265 raft_consensus.cc:729] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3070d2261747659b8c90867bd9ae36, State: Initialized, Role: FOLLOWER
I20260924 04:49:38.356370 16264 raft_consensus.cc:348] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:38.356675 16264 raft_consensus.cc:374] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:38.356526 16266 raft_consensus.cc:348] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:38.356753 16266 raft_consensus.cc:374] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:38.356752 16264 raft_consensus.cc:729] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e20568f2f5e74956bc62367868e4c8a4, State: Initialized, Role: FOLLOWER
I20260924 04:49:38.356808 16266 raft_consensus.cc:729] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f36a7e4b4ca463ab97ad261f3a5f0db, State: Initialized, Role: FOLLOWER
I20260924 04:49:38.357023 16265 consensus_queue.cc:260] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:38.357195 16266 consensus_queue.cc:260] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:38.357195 16264 consensus_queue.cc:260] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:38.357916 16113 heartbeater.cc:503] Master 127.0.98.62:34241 was elected leader, sending a full tablet report...
I20260924 04:49:38.358134 16265 ts_tablet_manager.cc:1428] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:49:38.358173 16264 ts_tablet_manager.cc:1428] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260924 04:49:38.358192 16247 heartbeater.cc:503] Master 127.0.98.62:34241 was elected leader, sending a full tablet report...
I20260924 04:49:38.359051 15979 heartbeater.cc:503] Master 127.0.98.62:34241 was elected leader, sending a full tablet report...
I20260924 04:49:38.360343 16266 ts_tablet_manager.cc:1428] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 04:49:38.481437 16270 raft_consensus.cc:482] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:49:38.481648 16270 raft_consensus.cc:504] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:38.482903 16270 leader_election.cc:288] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657), e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755)
I20260924 04:49:38.487106 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.483375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54057 (local address 127.0.98.2:42657)
0924 04:49:38.483511 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:38.483516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:38.483783 (+ 267us) server_negotiation.cc:407] Connection header received
0924 04:49:38.484018 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:38.484021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:38.484071 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:38.484139 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:38.484460 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.484955 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.486119 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.486534 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.486628 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.486699 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:38.486883 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 04:49:38.486958 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:49:38.487138 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.483138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42657 (local address 127.0.98.1:54057)
0924 04:49:38.483693 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:38.483708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:38.483862 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:38.484216 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:38.484219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:38.484236 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:38.484355 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:38.484360 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.485058 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.485061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:38.485998 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:38.486006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.486651 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.486654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:38.486768 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.486790 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:49:38.486851 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 04:49:38.486924 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":63,"spinlock_wait_cycles":15104,"thread_start_us":159,"threads_started":1}
I20260924 04:49:38.487154 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.483383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47777 (local address 127.0.98.3:42755)
0924 04:49:38.483511 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:38.483516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:38.483891 (+ 375us) server_negotiation.cc:407] Connection header received
0924 04:49:38.484096 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:38.484099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:38.484145 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:38.484210 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:38.484494 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.484968 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.486123 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.486529 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.486628 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.486692 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:38.486939 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 04:49:38.487036 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:49:38.487344 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.483159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42755 (local address 127.0.98.1:47777)
0924 04:49:38.483804 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:38.483817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:38.483988 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:38.484238 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:38.484241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:38.484257 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:38.484386 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:38.484391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.485070 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.485074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:38.485996 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:38.486005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.486650 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.486653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:38.486822 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.486840 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:49:38.486904 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:49:38.487003 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":204,"thread_start_us":132,"threads_started":1}
I20260924 04:49:38.487870 16067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78fe5d52fca44dd8b4d715a183a8d32" candidate_uuid: "9b3070d2261747659b8c90867bd9ae36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" is_pre_election: true
I20260924 04:49:38.487963 16201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78fe5d52fca44dd8b4d715a183a8d32" candidate_uuid: "9b3070d2261747659b8c90867bd9ae36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e20568f2f5e74956bc62367868e4c8a4" is_pre_election: true
I20260924 04:49:38.488225 16201 raft_consensus.cc:2454] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b3070d2261747659b8c90867bd9ae36 in term 0.
I20260924 04:49:38.488226 16067 raft_consensus.cc:2454] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b3070d2261747659b8c90867bd9ae36 in term 0.
I20260924 04:49:38.488725 15868 leader_election.cc:302] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b3070d2261747659b8c90867bd9ae36, e20568f2f5e74956bc62367868e4c8a4; no voters:
I20260924 04:49:38.489018 16270 raft_consensus.cc:2781] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:49:38.489100 16270 raft_consensus.cc:482] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:49:38.489151 16270 raft_consensus.cc:3037] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:38.490250 16270 raft_consensus.cc:504] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:38.490617 16270 leader_election.cc:288] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 1 election: Requested vote from peers 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657), e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755)
I20260924 04:49:38.491031 16067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78fe5d52fca44dd8b4d715a183a8d32" candidate_uuid: "9b3070d2261747659b8c90867bd9ae36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db"
I20260924 04:49:38.491027 16201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78fe5d52fca44dd8b4d715a183a8d32" candidate_uuid: "9b3070d2261747659b8c90867bd9ae36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e20568f2f5e74956bc62367868e4c8a4"
I20260924 04:49:38.491148 16201 raft_consensus.cc:3037] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:38.491170 16067 raft_consensus.cc:3037] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:38.492188 16201 raft_consensus.cc:2454] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b3070d2261747659b8c90867bd9ae36 in term 1.
I20260924 04:49:38.492252 16067 raft_consensus.cc:2454] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b3070d2261747659b8c90867bd9ae36 in term 1.
I20260924 04:49:38.492545 15867 leader_election.cc:302] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f36a7e4b4ca463ab97ad261f3a5f0db, 9b3070d2261747659b8c90867bd9ae36; no voters:
I20260924 04:49:38.492769 16270 raft_consensus.cc:2781] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:49:38.493005 16270 raft_consensus.cc:686] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 1 LEADER]: Becoming Leader. State: Replica: 9b3070d2261747659b8c90867bd9ae36, State: Running, Role: LEADER
I20260924 04:49:38.493371 16270 consensus_queue.cc:237] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:38.496562 15788 catalog_manager.cc:5495] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b3070d2261747659b8c90867bd9ae36 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9b3070d2261747659b8c90867bd9ae36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } health_report { overall_health: HEALTHY } } }
I20260924 04:49:38.529561 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.525867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.0.98.1:42927)
0924 04:49:38.526004 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:38.526014 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:38.526032 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:49:38.526216 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:38.526219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:38.526279 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:38.526367 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:49:38.526624 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.527213 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.528068 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:38.528280 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:38.528368 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:38.528716 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:49:38.529140 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:49:38.529249 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:38.529352 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 04:49:38.529419 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 04:49:38.535262 16248 tablet.cc:2369] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:49:38.536441 16201 raft_consensus.cc:1261] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4 [term 1 FOLLOWER]: Refusing update from remote peer 9b3070d2261747659b8c90867bd9ae36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:49:38.536550 16067 raft_consensus.cc:1261] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 1 FOLLOWER]: Refusing update from remote peer 9b3070d2261747659b8c90867bd9ae36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:49:38.537228 16270 consensus_queue.cc:1035] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:49:38.537468 16275 consensus_queue.cc:1035] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:49:38.546550 16279 mvcc.cc:205] Tried to move back new op lower bound from 7332763150476877824 to 7332763150311518208. Current Snapshot: MvccSnapshot[applied={T|T < 7332763150476877824}]
I20260924 04:49:38.555883 15787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:49:38.556061 15787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:49:38.558634 15787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Sending DeleteTablet for 3 replicas of tablet c78fe5d52fca44dd8b4d715a183a8d32
I20260924 04:49:38.559491 16047 tablet_service.cc:1514] Processing DeleteTablet for tablet c78fe5d52fca44dd8b4d715a183a8d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37038
I20260924 04:49:38.559569 15913 tablet_service.cc:1514] Processing DeleteTablet for tablet c78fe5d52fca44dd8b4d715a183a8d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56894
I20260924 04:49:38.559700 16181 tablet_service.cc:1514] Processing DeleteTablet for tablet c78fe5d52fca44dd8b4d715a183a8d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52326
I20260924 04:49:38.560220 16290 tablet_replica.cc:330] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4: stopping tablet replica
I20260924 04:49:38.560464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 15850
I20260924 04:49:38.560458 16288 tablet_replica.cc:330] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db: stopping tablet replica
I20260924 04:49:38.561110 16290 raft_consensus.cc:2229] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:49:38.561501 16290 raft_consensus.cc:2258] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:49:38.561849 16288 raft_consensus.cc:2229] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:49:38.562170 16288 raft_consensus.cc:2258] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:49:38.563298 16290 ts_tablet_manager.cc:1905] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:49:38.563333 16288 ts_tablet_manager.cc:1905] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:49:38.565523 16288 ts_tablet_manager.cc:1918] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:49:38.565623 16288 log.cc:1196] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/c78fe5d52fca44dd8b4d715a183a8d32
I20260924 04:49:38.565881 16288 ts_tablet_manager.cc:1939] T c78fe5d52fca44dd8b4d715a183a8d32 P 5f36a7e4b4ca463ab97ad261f3a5f0db: Deleting consensus metadata
I20260924 04:49:38.565929 16290 ts_tablet_manager.cc:1918] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:49:38.566025 16290 log.cc:1196] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/c78fe5d52fca44dd8b4d715a183a8d32
I20260924 04:49:38.566323 16290 ts_tablet_manager.cc:1939] T c78fe5d52fca44dd8b4d715a183a8d32 P e20568f2f5e74956bc62367868e4c8a4: Deleting consensus metadata
I20260924 04:49:38.567168 15773 catalog_manager.cc:4841] TS 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657): tablet c78fe5d52fca44dd8b4d715a183a8d32 (table test-table [id=9e9d2cb4feab46fa8e9fe00c29a79768]) successfully deleted
I20260924 04:49:38.567433 15774 catalog_manager.cc:4841] TS e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755): tablet c78fe5d52fca44dd8b4d715a183a8d32 (table test-table [id=9e9d2cb4feab46fa8e9fe00c29a79768]) successfully deleted
W20260924 04:49:38.567762 15774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42927 (error 108)
I20260924 04:49:38.567999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 15984
I20260924 04:49:38.569180 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:38.568896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42927 (local address 127.0.0.1:56922)
0924 04:49:38.569030 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:38.569105 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 04:49:38.569325 15774 catalog_manager.cc:4584] TS 9b3070d2261747659b8c90867bd9ae36 (127.0.98.1:42927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c78fe5d52fca44dd8b4d715a183a8d32: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42927: connect: Connection refused (error 111)
I20260924 04:49:38.573030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 16118
I20260924 04:49:38.577612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 15756
I20260924 04:49:58.583382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34241
--webserver_interface=127.0.98.62
--webserver_port=42435
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:49:58.693863 16292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:58.694159 16292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:58.694201 16292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:58.698328 16292 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:49:58.698412 16292 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:49:58.698427 16292 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:49:58.698441 16292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:49:58.698457 16292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:58.698474 16292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:58.698488 16292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:49:58.698503 16292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:49:58.704034 16292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:58.705153 16292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:58.706249 16292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:58.719446 16298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:58.719404 16301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:58.719482 16292 server_base.cc:1034] running on GCE node
W20260924 04:49:58.719404 16299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:58.720270 16292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:58.721470 16292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:58.722630 16292 hybrid_clock.cc:648] HybridClock initialized: now 1790225398722601 us; error 46 us; skew 500 ppm
Sep 24 04:49:58 dist-test-slave-hvcl krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:58.725353 16292 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:49:58.726303 16292 webserver.cc:460] Webserver started at http://127.0.98.62:42435/ using document root <none> and password file <none>
I20260924 04:49:58.726881 16292 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:58.726960 16292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:58.731282 16292 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 04:49:58.733577 16308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:58.734866 16292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:49:58.735009 16292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "ebcdcf67db39499293d8a608a5cb4a79"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "ade310c1610f57b6549f70ad1bff6cd3"
server_key_iv: "00ddd90136f7b50cb3732425f1320504"
server_key_version: "encryptionkey@0"
I20260924 04:49:58.735666 16292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:58.750532 16292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:58.753902 16292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:58.754088 16292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:58.761040 16292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34241
I20260924 04:49:58.761293 16360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34241 every 8 connection(s)
I20260924 04:49:58.762174 16292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:49:58.766553 16361 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:49:58.768289 16361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Bootstrap starting.
I20260924 04:49:58.770145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 16292
I20260924 04:49:58.770591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42927
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34241
--webserver_port=36721
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:49:58.785501 16361 log.cc:824] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:58.794500 16361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:49:58.794864 16361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Bootstrap complete.
I20260924 04:49:58.798818 16361 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 04:49:58.799328 16361 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcdcf67db39499293d8a608a5cb4a79, State: Initialized, Role: FOLLOWER
I20260924 04:49:58.799825 16361 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 04:49:58.799952 16361 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:49:58.800024 16361 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:49:58.800118 16361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:49:58.802117 16361 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 04:49:58.802460 16361 leader_election.cc:302] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebcdcf67db39499293d8a608a5cb4a79; no voters:
I20260924 04:49:58.802783 16361 leader_election.cc:288] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:49:58.802884 16365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:49:58.803135 16365 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 2 LEADER]: Becoming Leader. State: Replica: ebcdcf67db39499293d8a608a5cb4a79, State: Running, Role: LEADER
I20260924 04:49:58.803444 16365 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 04:49:58.804140 16361 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:49:58.804952 16366 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ebcdcf67db39499293d8a608a5cb4a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } } }
I20260924 04:49:58.805078 16366 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:58.805500 16367 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebcdcf67db39499293d8a608a5cb4a79. Latest consensus state: current_term: 2 leader_uuid: "ebcdcf67db39499293d8a608a5cb4a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } } }
I20260924 04:49:58.805635 16367 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: This master's current role is: LEADER
I20260924 04:49:58.805918 16374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:49:58.811225 16374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:49:58.811929 16374 catalog_manager.cc:1259] Loaded cluster ID: f32e0c94c01e4ae49615adfb3984a307
I20260924 04:49:58.812000 16374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:49:58.814512 16374 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:49:58.815357 16374 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Loaded TSK: 0
I20260924 04:49:58.831095 16374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Generated new TSK 1
I20260924 04:49:58.831418 16374 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:49:58.890730 16363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:58.891077 16363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:58.891129 16363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:58.895253 16363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:58.895319 16363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:58.895386 16363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:49:58.900928 16363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36721
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:58.902021 16363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:58.903074 16363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:58.912782 16388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:58.913035 16391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:58.912782 16389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:58.913107 16363 server_base.cc:1034] running on GCE node
I20260924 04:49:58.913661 16363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:58.914327 16363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:58.915490 16363 hybrid_clock.cc:648] HybridClock initialized: now 1790225398915460 us; error 47 us; skew 500 ppm
Sep 24 04:49:58 dist-test-slave-hvcl krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:58.918120 16363 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:49:58.919029 16363 webserver.cc:460] Webserver started at http://127.0.98.1:36721/ using document root <none> and password file <none>
I20260924 04:49:58.919565 16363 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:58.919634 16363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:58.923614 16363 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:49:58.925854 16398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:58.927225 16363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:49:58.927333 16363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "9b3070d2261747659b8c90867bd9ae36"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "043bca0ce576c43476cdab4a3ce52d37"
server_key_iv: "809d7e5fbaa6be9b4446099daceecaca"
server_key_version: "encryptionkey@0"
I20260924 04:49:58.927897 16363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:58.939731 16363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:58.942935 16363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:58.943172 16363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:58.943768 16363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:58.944859 16405 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:49:58.949115 16363 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:49:58.949203 16363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 04:49:58.949287 16363 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:49:58.951164 16363 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:49:58.951217 16363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260924 04:49:58.951524 16405 tablet_bootstrap.cc:492] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: Bootstrap starting.
I20260924 04:49:58.961038 16363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42927
I20260924 04:49:58.961115 16512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42927 every 8 connection(s)
I20260924 04:49:58.962019 16363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:49:58.967099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 16363
I20260924 04:49:58.967641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42657
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34241
--webserver_port=41753
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:49:58.969743 16405 log.cc:824] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:49:58 dist-test-slave-hvcl krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:58.977946 16405 tablet_bootstrap.cc:492] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:49:58.978374 16405 tablet_bootstrap.cc:492] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: Bootstrap complete.
I20260924 04:49:58.979470 16405 ts_tablet_manager.cc:1397] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: Time spent bootstrapping tablet: real 0.028s user 0.019s sys 0.008s
I20260924 04:49:58.980561 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:58.964255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46429 (local address 127.0.98.62:34241)
0924 04:49:58.964677 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:58.964700 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:58.965006 (+ 306us) server_negotiation.cc:407] Connection header received
0924 04:49:58.966120 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:58.966131 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:58.966439 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:58.966758 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:58.968090 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:58.969221 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:58.970795 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:58.971544 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:58.971735 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:58.974742 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:58.974749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:58.974767 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:58.974799 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:58.976852 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:58.977483 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:58.977489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:58.977495 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:58.977661 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:58.978089 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:58.978092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:58.978094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:58.978593 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:58.978727 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:58.979371 (+ 644us) server_negotiation.cc:299] Negotiation successful
0924 04:49:58.979707 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":134,"threads_started":1}
I20260924 04:49:58.980587 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:58.964257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:46429)
0924 04:49:58.964837 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:58.964880 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:58.965697 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:58.966838 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:58.966846 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:58.967357 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:58.967908 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:58.967929 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:58.969396 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:58.969400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:58.970662 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:58.970674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:58.971704 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:58.971708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:58.971925 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:58.972608 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:58.972630 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:58.974517 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:58.976984 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:58.976991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:58.977005 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:58.977336 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:58.977766 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:58.977770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:58.977771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:58.977943 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:58.978708 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:58.978715 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:58.979038 (+ 323us) client_negotiation.cc:769] Sending connection context
0924 04:49:58.979226 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 04:49:58.979627 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":142,"threads_started":1}
I20260924 04:49:58.982280 16405 raft_consensus.cc:348] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:58.982822 16405 raft_consensus.cc:729] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3070d2261747659b8c90867bd9ae36, State: Initialized, Role: FOLLOWER
I20260924 04:49:58.983359 16405 consensus_queue.cc:260] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:58.983834 16513 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34241
I20260924 04:49:58.984056 16513 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:58.984449 16405 ts_tablet_manager.cc:1428] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:49:58.984629 16513 heartbeater.cc:511] Master 127.0.98.62:34241 requested a full tablet report, sending...
I20260924 04:49:58.987175 16325 ts_manager.cc:194] Registered new tserver with Master: 9b3070d2261747659b8c90867bd9ae36 (127.0.98.1:42927)
I20260924 04:49:58.990604 16325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46429
I20260924 04:49:58.993284 16513 heartbeater.cc:503] Master 127.0.98.62:34241 was elected leader, sending a full tablet report...
Sep 24 04:49:58 dist-test-slave-hvcl krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:49:59.000700 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:58.990090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42927 (local address 127.0.0.1:43808)
0924 04:49:58.990868 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:58.990912 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:58.991113 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:58.991639 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:58.991645 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:58.992119 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:58.992253 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:58.992262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:58.993586 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:58.993590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:58.994303 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:58.994314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:58.994996 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:58.995012 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:58.995120 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:58.995669 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:58.995680 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:58.997161 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:58.999111 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:58.999119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:58.999123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:58.999347 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:58.999626 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:58.999628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:58.999630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:58.999705 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:59.000162 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:59.000167 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:59.000269 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 04:49:59.000378 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 04:49:59.000535 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":118,"threads_started":1}
I20260924 04:49:59.000801 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:58.990216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43808 (local address 127.0.98.1:42927)
0924 04:49:58.990604 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:58.990616 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:58.991051 (+ 435us) server_negotiation.cc:407] Connection header received
0924 04:49:58.991268 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:58.991273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:58.991486 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:58.991621 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:58.992386 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:58.993465 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:58.994414 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:58.994761 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:58.994859 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:58.997305 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:58.997316 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:58.997329 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:58.997356 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:58.998998 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:58.999449 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:58.999455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:58.999457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:58.999547 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:58.999777 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:58.999780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:58.999781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:59.000070 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:59.000191 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:59.000473 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 04:49:59.000640 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":93,"threads_started":1}
I20260924 04:49:59.001771 16447 tablet_service.cc:1514] Processing DeleteTablet for tablet c78fe5d52fca44dd8b4d715a183a8d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43808
I20260924 04:49:59.002664 16522 tablet_replica.cc:330] stopping tablet replica
I20260924 04:49:59.003038 16522 raft_consensus.cc:2229] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:49:59.003324 16522 raft_consensus.cc:2258] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:49:59.004527 16522 ts_tablet_manager.cc:1905] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:49:59.009212 16522 ts_tablet_manager.cc:1918] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:49:59.009330 16522 log.cc:1196] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal/wals/c78fe5d52fca44dd8b4d715a183a8d32
I20260924 04:49:59.009594 16522 ts_tablet_manager.cc:1939] T c78fe5d52fca44dd8b4d715a183a8d32 P 9b3070d2261747659b8c90867bd9ae36: Deleting consensus metadata
I20260924 04:49:59.011691 16311 catalog_manager.cc:4841] TS 9b3070d2261747659b8c90867bd9ae36 (127.0.98.1:42927): tablet c78fe5d52fca44dd8b4d715a183a8d32 (table test-table [id=9e9d2cb4feab46fa8e9fe00c29a79768]) successfully deleted
W20260924 04:49:59.088903 16517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:59.089187 16517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:59.089231 16517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:59.093240 16517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:59.093307 16517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:59.093369 16517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:49:59.098636 16517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41753
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:59.099674 16517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:59.100656 16517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:59.110158 16528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:59.110153 16527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:59.110313 16517 server_base.cc:1034] running on GCE node
W20260924 04:49:59.110227 16530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:59.110867 16517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:59.111584 16517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:59.112739 16517 hybrid_clock.cc:648] HybridClock initialized: now 1790225399112702 us; error 51 us; skew 500 ppm
Sep 24 04:49:59 dist-test-slave-hvcl krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:59.115644 16517 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:49:59.116586 16517 webserver.cc:460] Webserver started at http://127.0.98.2:41753/ using document root <none> and password file <none>
I20260924 04:49:59.117170 16517 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:59.117244 16517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:59.121129 16517 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:49:59.123299 16537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:59.124646 16517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:49:59.124754 16517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "ccc980eb97cacd6345412a992b1b028f"
server_key_iv: "eda93f12a8b122c18bc8365cb12a4118"
server_key_version: "encryptionkey@0"
I20260924 04:49:59.125311 16517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:59.143229 16517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:59.146485 16517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:59.146652 16517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:59.147253 16517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:59.148072 16517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:59.148115 16517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:59.148154 16517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:59.148172 16517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:59.156589 16517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42657
I20260924 04:49:59.156621 16650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42657 every 8 connection(s)
I20260924 04:49:59.157645 16517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:49:59.164572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 16517
I20260924 04:49:59.165036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42755
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34241
--webserver_port=45287
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:49:59 dist-test-slave-hvcl krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:59.171818 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.159562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43207 (local address 127.0.98.62:34241)
0924 04:49:59.159696 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:59.159699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:59.160772 (+ 1073us) server_negotiation.cc:407] Connection header received
0924 04:49:59.161701 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:59.161705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:59.161766 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:59.161840 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:59.163138 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.163665 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.164866 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.165225 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.165332 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.168066 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:59.168069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:59.168071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:59.168100 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:59.169489 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:59.170110 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:59.170113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:59.170114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:59.170190 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:59.170600 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:59.170603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:59.170605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:59.170806 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:59.170893 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:59.171569 (+ 676us) server_negotiation.cc:299] Negotiation successful
0924 04:49:59.171690 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:49:59.173102 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.159961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:43207)
0924 04:49:59.160619 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:59.160662 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:59.161471 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:59.162035 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:59.162043 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:59.162418 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:59.162941 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:59.162955 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.163792 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.163795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:59.164711 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:59.164723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.165372 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.165375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:59.165523 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.166133 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:59.166150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:59.167849 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:59.169607 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:59.169614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:59.169632 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:59.170000 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:59.170272 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:59.170275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:59.170276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:59.170483 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:59.170911 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:59.170918 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:59.171307 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 04:49:59.171543 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 04:49:59.171940 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":109,"threads_started":1}
I20260924 04:49:59.174321 16651 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34241
I20260924 04:49:59.174603 16651 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:59.175266 16651 heartbeater.cc:511] Master 127.0.98.62:34241 requested a full tablet report, sending...
I20260924 04:49:59.176367 16325 ts_manager.cc:194] Registered new tserver with Master: 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657)
I20260924 04:49:59.176949 16325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43207
W20260924 04:49:59.278805 16654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:49:59.279137 16654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:49:59.279193 16654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:49:59.283252 16654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:49:59.283322 16654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:49:59.283388 16654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:49:59.288784 16654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45287
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:49:59.289875 16654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:49:59.290920 16654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:49:59.300444 16659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:59.300472 16662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:49:59.300510 16660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:49:59.300689 16654 server_base.cc:1034] running on GCE node
I20260924 04:49:59.301173 16654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:49:59.301836 16654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:49:59.303009 16654 hybrid_clock.cc:648] HybridClock initialized: now 1790225399302992 us; error 27 us; skew 500 ppm
Sep 24 04:49:59 dist-test-slave-hvcl krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:49:59.306036 16654 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:49:59.307014 16654 webserver.cc:460] Webserver started at http://127.0.98.3:45287/ using document root <none> and password file <none>
I20260924 04:49:59.307562 16654 fs_manager.cc:359] Metadata directory not provided
I20260924 04:49:59.307644 16654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:49:59.311697 16654 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:49:59.313920 16669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:59.315315 16654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:49:59.315447 16654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "e20568f2f5e74956bc62367868e4c8a4"
format_stamp: "Formatted at 2026-09-24 04:49:38 on dist-test-slave-hvcl"
server_key: "a2e28863cc31bb4ebb9f92e78de14743"
server_key_iv: "61ed705fcfaf6e88a217df84889830cc"
server_key_version: "encryptionkey@0"
I20260924 04:49:59.316006 16654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:49:59.339900 16654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:49:59.343520 16654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:49:59.343712 16654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:49:59.344347 16654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:49:59.345211 16654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:49:59.345258 16654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:59.345301 16654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:49:59.345329 16654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:49:59.353957 16654 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42755
I20260924 04:49:59.353996 16782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42755 every 8 connection(s)
I20260924 04:49:59.354948 16654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:49:59.361989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 16654
Sep 24 04:49:59 dist-test-slave-hvcl krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:49:59.369004 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.356904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50463 (local address 127.0.98.62:34241)
0924 04:49:59.357059 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:59.357064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:59.358216 (+ 1152us) server_negotiation.cc:407] Connection header received
0924 04:49:59.359171 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:59.359174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:59.359223 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:59.359305 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:59.360597 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.361130 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.362412 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.362692 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.362789 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.365430 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:59.365434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:59.365436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:59.365459 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:59.366843 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:59.367373 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:59.367377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:59.367378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:59.367455 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:59.367810 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:59.367813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:59.367814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:59.367975 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:59.368053 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:59.368641 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 04:49:59.368789 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:49:59.369843 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.357368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:50463)
0924 04:49:59.358058 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:59.358100 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:59.358941 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:59.359493 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:59.359500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:59.359912 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:49:59.360422 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:59.360440 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.361301 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.361305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:59.362281 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:59.362292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.362852 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.362857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:59.363087 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.363652 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:49:59.363668 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:49:59.365261 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:49:59.366953 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:59.366959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:59.367007 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:59.367280 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:59.367536 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:49:59.367539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:49:59.367541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:49:59.367714 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:49:59.368062 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:49:59.368067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:49:59.368345 (+ 278us) client_negotiation.cc:769] Sending connection context
0924 04:49:59.368560 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 04:49:59.368932 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":123,"threads_started":1}
I20260924 04:49:59.370498 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.362725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.0.98.62:34241)
0924 04:49:59.363142 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:59.363146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:59.363247 (+ 101us) server_negotiation.cc:407] Connection header received
0924 04:49:59.363382 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:59.363384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:59.363428 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:59.363501 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:59.364210 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.365002 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.365819 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.366049 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.366133 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.367403 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:59.367406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:59.367409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:59.367435 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:59.368984 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:59.369389 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:59.369392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:59.369394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:59.369474 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:59.369713 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:59.369716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:59.369718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:59.369881 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:59.370067 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:59.370235 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 04:49:59.370347 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":83,"threads_started":1}
I20260924 04:49:59.371055 16783 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34241
I20260924 04:49:59.371284 16783 heartbeater.cc:464] Registering TS with master...
I20260924 04:49:59.371788 16783 heartbeater.cc:511] Master 127.0.98.62:34241 requested a full tablet report, sending...
I20260924 04:49:59.372745 16325 ts_manager.cc:194] Registered new tserver with Master: e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755)
I20260924 04:49:59.373274 16325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50463
I20260924 04:49:59.384047 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:49:59.388156 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.384645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.0.98.62:34241)
0924 04:49:59.384767 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:59.384772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:59.384952 (+ 180us) server_negotiation.cc:407] Connection header received
0924 04:49:59.385170 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:59.385172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:59.385281 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:59.385372 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:49:59.385600 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.386113 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.386996 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.387232 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.387321 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.387620 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:49:59.387766 (+ 146us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 04:49:59.388019 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37118: token expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 04:49:59.388226 16787 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37118: token expired
W20260924 04:49:59.388650 16251 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:34241)
I20260924 04:49:59.395413 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.389006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.0.98.62:34241)
0924 04:49:59.389119 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:59.389122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:59.389243 (+ 121us) server_negotiation.cc:407] Connection header received
0924 04:49:59.389328 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:59.389330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:59.389367 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:59.389427 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:49:59.389975 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.390473 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.391345 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.391553 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.391629 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.392864 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:49:59.392867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:49:59.392869 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:49:59.392888 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:49:59.394101 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:59.394456 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:59.394459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:59.394460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:59.394529 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:49:59.394733 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:49:59.394735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:49:59.394736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:49:59.394897 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:49:59.395065 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:59.395196 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 04:49:59.395289 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:49:59.397857 16325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:49:59.401135 16325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:49:59.409389 16447 tablet_service.cc:1467] Processing CreateTablet for tablet edc3f7c89410426bab8d751c9028b56d (DEFAULT_TABLE table=test-table [id=a22d0c215a6941b894469e8ec0110655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:59.410094 16447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc3f7c89410426bab8d751c9028b56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:59.412846 16405 tablet_bootstrap.cc:492] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36: Bootstrap starting.
I20260924 04:49:59.413791 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.408161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42657 (local address 127.0.0.1:55438)
0924 04:49:59.408329 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:59.408344 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:59.408439 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:59.409285 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:59.409288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:59.409306 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:59.409432 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:59.409440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.410919 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.410922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:59.412259 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:59.412268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.413319 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.413323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:59.413430 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.413479 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 04:49:59.413558 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:49:59.413651 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":7}
I20260924 04:49:59.413836 16405 tablet_bootstrap.cc:654] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:59.414589 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.408345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42755 (local address 127.0.0.1:43380)
0924 04:49:59.408902 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:59.408916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:59.409011 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:59.410069 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:59.410072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:59.410089 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:59.410299 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:59.410306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.412038 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.412042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:59.412943 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:59.412954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.414021 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.414024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:59.414113 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.414138 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:49:59.414370 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 04:49:59.414452 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":23,"thread_start_us":101,"threads_started":1}
I20260924 04:49:59.414673 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.408498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.0.98.3:42755)
0924 04:49:59.409621 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:59.409633 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:59.409654 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:49:59.409727 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:59.409733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:59.409931 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:59.410085 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:59.410403 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.411900 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.413091 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.413896 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.414023 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.414185 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:59.414416 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:49:59.414527 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1015,"thread_start_us":118,"threads_started":1}
I20260924 04:49:59.415126 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.408498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.0.98.2:42657)
0924 04:49:59.408821 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:59.408834 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:59.408857 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:49:59.408945 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:59.408949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:59.409143 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:59.409296 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:59.409579 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.410671 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.412391 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.413198 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.413304 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.413481 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:59.414854 (+ 1373us) server_negotiation.cc:299] Negotiation successful
0924 04:49:59.414959 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":106,"threads_started":1}
I20260924 04:49:59.415776 16405 tablet_bootstrap.cc:492] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36: No bootstrap required, opened a new log
I20260924 04:49:59.415874 16405 ts_tablet_manager.cc:1397] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260924 04:49:59.416338 16405 raft_consensus.cc:348] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:59.416481 16405 raft_consensus.cc:374] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:59.416529 16405 raft_consensus.cc:729] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3070d2261747659b8c90867bd9ae36, State: Initialized, Role: FOLLOWER
I20260924 04:49:59.416706 16405 consensus_queue.cc:260] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:59.416990 16405 ts_tablet_manager.cc:1428] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:49:59.417927 16717 tablet_service.cc:1467] Processing CreateTablet for tablet edc3f7c89410426bab8d751c9028b56d (DEFAULT_TABLE table=test-table [id=a22d0c215a6941b894469e8ec0110655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:59.417943 16585 tablet_service.cc:1467] Processing CreateTablet for tablet edc3f7c89410426bab8d751c9028b56d (DEFAULT_TABLE table=test-table [id=a22d0c215a6941b894469e8ec0110655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:49:59.419044 16585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc3f7c89410426bab8d751c9028b56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:59.419046 16717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc3f7c89410426bab8d751c9028b56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:49:59.426599 16796 tablet_bootstrap.cc:492] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4: Bootstrap starting.
I20260924 04:49:59.427140 16797 tablet_bootstrap.cc:492] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db: Bootstrap starting.
I20260924 04:49:59.429001 16796 tablet_bootstrap.cc:654] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:59.429358 16797 tablet_bootstrap.cc:654] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db: Neither blocks nor log segments found. Creating new log.
I20260924 04:49:59.430076 16797 log.cc:824] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:59.430090 16796 log.cc:824] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:49:59.432049 16797 tablet_bootstrap.cc:492] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db: No bootstrap required, opened a new log
I20260924 04:49:59.432073 16796 tablet_bootstrap.cc:492] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4: No bootstrap required, opened a new log
I20260924 04:49:59.432271 16796 ts_tablet_manager.cc:1397] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:49:59.432272 16797 ts_tablet_manager.cc:1397] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:49:59.435437 16797 raft_consensus.cc:348] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:59.435437 16796 raft_consensus.cc:348] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:59.435712 16797 raft_consensus.cc:374] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:59.435714 16796 raft_consensus.cc:374] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:49:59.435746 16797 raft_consensus.cc:729] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f36a7e4b4ca463ab97ad261f3a5f0db, State: Initialized, Role: FOLLOWER
I20260924 04:49:59.435758 16796 raft_consensus.cc:729] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e20568f2f5e74956bc62367868e4c8a4, State: Initialized, Role: FOLLOWER
I20260924 04:49:59.436172 16797 consensus_queue.cc:260] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:59.436184 16796 consensus_queue.cc:260] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:59.436873 16651 heartbeater.cc:503] Master 127.0.98.62:34241 was elected leader, sending a full tablet report...
I20260924 04:49:59.437126 16783 heartbeater.cc:503] Master 127.0.98.62:34241 was elected leader, sending a full tablet report...
I20260924 04:49:59.437187 16796 ts_tablet_manager.cc:1428] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:49:59.438125 16797 ts_tablet_manager.cc:1428] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 04:49:59.463222 16514 tablet.cc:2369] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:49:59.485371 16519 raft_consensus.cc:482] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:49:59.485626 16519 raft_consensus.cc:504] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:59.486678 16519 leader_election.cc:288] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657), e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755)
I20260924 04:49:59.490698 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.487129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60777 (local address 127.0.98.2:42657)
0924 04:49:59.487281 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:59.487286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:59.487407 (+ 121us) server_negotiation.cc:407] Connection header received
0924 04:49:59.487603 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:59.487606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:59.487663 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:59.487732 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:59.488058 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.488696 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.489750 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.490160 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.490248 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.490310 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:59.490500 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 04:49:59.490572 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 04:49:59.490698 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.487089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44877 (local address 127.0.98.3:42755)
0924 04:49:59.487282 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:59.487286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:59.487386 (+ 100us) server_negotiation.cc:407] Connection header received
0924 04:49:59.487868 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:59.487871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:59.487918 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:59.487983 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:49:59.488233 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.488786 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.489775 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.490178 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.490268 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.490331 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:59.490497 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 04:49:59.490568 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 04:49:59.490731 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.486933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42657 (local address 127.0.98.1:60777)
0924 04:49:59.487337 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:59.487348 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:59.487459 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:59.487768 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:59.487772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:59.487792 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:59.487900 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:59.487905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.488800 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.488804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:59.489629 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:59.489638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.490271 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.490274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:59.490367 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.490407 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 04:49:59.490470 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:49:59.490564 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":50,"spinlock_wait_cycles":1280,"thread_start_us":82,"threads_started":1}
I20260924 04:49:59.490742 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.486933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42755 (local address 127.0.98.1:44877)
0924 04:49:59.487280 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:59.487297 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:49:59.487459 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:49:59.488018 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:49:59.488020 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:49:59.488037 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:49:59.488135 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:59.488140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.488890 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.488894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:59.489660 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:49:59.489670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.490301 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.490305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:49:59.490393 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.490410 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:49:59.490472 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:49:59.490566 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":6,"spinlock_wait_cycles":9088,"thread_start_us":84,"threads_started":1}
I20260924 04:49:59.491397 16737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc3f7c89410426bab8d751c9028b56d" candidate_uuid: "9b3070d2261747659b8c90867bd9ae36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e20568f2f5e74956bc62367868e4c8a4" is_pre_election: true
I20260924 04:49:59.491521 16605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc3f7c89410426bab8d751c9028b56d" candidate_uuid: "9b3070d2261747659b8c90867bd9ae36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" is_pre_election: true
I20260924 04:49:59.491724 16737 raft_consensus.cc:2454] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b3070d2261747659b8c90867bd9ae36 in term 0.
I20260924 04:49:59.491782 16605 raft_consensus.cc:2454] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b3070d2261747659b8c90867bd9ae36 in term 0.
I20260924 04:49:59.492234 16401 leader_election.cc:302] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b3070d2261747659b8c90867bd9ae36, e20568f2f5e74956bc62367868e4c8a4; no voters:
I20260924 04:49:59.492514 16519 raft_consensus.cc:2781] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:49:59.492594 16519 raft_consensus.cc:482] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:49:59.492625 16519 raft_consensus.cc:3037] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:59.493667 16519 raft_consensus.cc:504] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:59.494015 16519 leader_election.cc:288] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 1 election: Requested vote from peers 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657), e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755)
I20260924 04:49:59.494356 16605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc3f7c89410426bab8d751c9028b56d" candidate_uuid: "9b3070d2261747659b8c90867bd9ae36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db"
I20260924 04:49:59.494356 16737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc3f7c89410426bab8d751c9028b56d" candidate_uuid: "9b3070d2261747659b8c90867bd9ae36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e20568f2f5e74956bc62367868e4c8a4"
I20260924 04:49:59.494482 16605 raft_consensus.cc:3037] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:59.494482 16737 raft_consensus.cc:3037] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:49:59.495815 16737 raft_consensus.cc:2454] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b3070d2261747659b8c90867bd9ae36 in term 1.
I20260924 04:49:59.495815 16605 raft_consensus.cc:2454] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b3070d2261747659b8c90867bd9ae36 in term 1.
I20260924 04:49:59.496141 16401 leader_election.cc:302] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5f36a7e4b4ca463ab97ad261f3a5f0db, 9b3070d2261747659b8c90867bd9ae36, e20568f2f5e74956bc62367868e4c8a4; no voters:
I20260924 04:49:59.496321 16519 raft_consensus.cc:2781] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:49:59.496551 16519 raft_consensus.cc:686] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 LEADER]: Becoming Leader. State: Replica: 9b3070d2261747659b8c90867bd9ae36, State: Running, Role: LEADER
I20260924 04:49:59.496871 16519 consensus_queue.cc:237] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:49:59.499858 16323 catalog_manager.cc:5495] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b3070d2261747659b8c90867bd9ae36 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9b3070d2261747659b8c90867bd9ae36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } health_report { overall_health: HEALTHY } } }
I20260924 04:49:59.515139 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.511642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.0.98.1:42927)
0924 04:49:59.511893 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 04:49:59.511898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:49:59.511911 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:49:59.511989 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:49:59.511992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:49:59.512049 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:49:59.512186 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:49:59.512408 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.512933 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.513725 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:49:59.513941 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:49:59.514026 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:49:59.514281 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:49:59.514729 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:49:59.514832 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:49:59.514902 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 04:49:59.514959 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260924 04:49:59.519171 16737 raft_consensus.cc:1261] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4 [term 1 FOLLOWER]: Refusing update from remote peer 9b3070d2261747659b8c90867bd9ae36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:49:59.519171 16605 raft_consensus.cc:1261] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 1 FOLLOWER]: Refusing update from remote peer 9b3070d2261747659b8c90867bd9ae36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:49:59.519953 16519 consensus_queue.cc:1035] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:49:59.520151 16804 consensus_queue.cc:1035] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:49:59.527931 16808 mvcc.cc:205] Tried to move back new op lower bound from 7332763236423372800 to 7332763236341952512. Current Snapshot: MvccSnapshot[applied={T|T < 7332763236423372800}]
I20260924 04:49:59.536053 16323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:49:59.536253 16323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:49:59.539896 16323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Sending DeleteTablet for 3 replicas of tablet edc3f7c89410426bab8d751c9028b56d
I20260924 04:49:59.540807 16447 tablet_service.cc:1514] Processing DeleteTablet for tablet edc3f7c89410426bab8d751c9028b56d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43808
I20260924 04:49:59.540762 16585 tablet_service.cc:1514] Processing DeleteTablet for tablet edc3f7c89410426bab8d751c9028b56d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55438
I20260924 04:49:59.540896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 16363
I20260924 04:49:59.541815 16717 tablet_service.cc:1514] Processing DeleteTablet for tablet edc3f7c89410426bab8d751c9028b56d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:49:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43380
I20260924 04:49:59.542284 16817 tablet_replica.cc:330] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4: stopping tablet replica
I20260924 04:49:59.542593 16816 tablet_replica.cc:330] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db: stopping tablet replica
I20260924 04:49:59.542658 16817 raft_consensus.cc:2229] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:49:59.542889 16816 raft_consensus.cc:2229] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:49:59.543058 16817 raft_consensus.cc:2258] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:49:59.543215 16816 raft_consensus.cc:2258] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:49:59.544541 16817 ts_tablet_manager.cc:1905] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:49:59.544867 16816 ts_tablet_manager.cc:1905] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:49:59.546831 16311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42927 (error 108)
I20260924 04:49:59.547308 16816 ts_tablet_manager.cc:1918] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:49:59.547308 16817 ts_tablet_manager.cc:1918] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:49:59.547408 16817 log.cc:1196] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/edc3f7c89410426bab8d751c9028b56d
I20260924 04:49:59.547408 16816 log.cc:1196] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/edc3f7c89410426bab8d751c9028b56d
I20260924 04:49:59.547742 16817 ts_tablet_manager.cc:1939] T edc3f7c89410426bab8d751c9028b56d P e20568f2f5e74956bc62367868e4c8a4: Deleting consensus metadata
I20260924 04:49:59.547740 16816 ts_tablet_manager.cc:1939] T edc3f7c89410426bab8d751c9028b56d P 5f36a7e4b4ca463ab97ad261f3a5f0db: Deleting consensus metadata
I20260924 04:49:59.548113 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.547852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42927 (local address 127.0.0.1:43838)
0924 04:49:59.547960 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:59.548026 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 04:49:59.548125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 16517
W20260924 04:49:59.551155 16311 catalog_manager.cc:4584] TS 9b3070d2261747659b8c90867bd9ae36 (127.0.98.1:42927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet edc3f7c89410426bab8d751c9028b56d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42927: connect: Connection refused (error 111)
I20260924 04:49:59.551764 16311 catalog_manager.cc:4841] TS e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755): tablet edc3f7c89410426bab8d751c9028b56d (table test-table [id=a22d0c215a6941b894469e8ec0110655]) successfully deleted
I20260924 04:49:59.553984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 16654
I20260924 04:49:59.555186 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:49:59.554162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42657 (local address 127.0.0.1:55440)
0924 04:49:59.554941 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0924 04:49:59.555092 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":705}
I20260924 04:49:59.559005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 16292
I20260924 04:50:19.565075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34241
--webserver_interface=127.0.98.62
--webserver_port=42435
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:19.679104 16819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:19.679392 16819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:19.679441 16819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:19.683597 16819 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:50:19.683662 16819 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:50:19.683676 16819 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:50:19.683689 16819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:19.683707 16819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:19.683725 16819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:19.683740 16819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:19.683754 16819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:19.689689 16819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:19.690855 16819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:19.691983 16819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:19.700721 16824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:19.700781 16825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:19.700865 16819 server_base.cc:1034] running on GCE node
W20260924 04:50:19.700810 16827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:19.701614 16819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:19.702821 16819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:19.703974 16819 hybrid_clock.cc:648] HybridClock initialized: now 1790225419703943 us; error 48 us; skew 500 ppm
Sep 24 04:50:19 dist-test-slave-hvcl krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:19.706882 16819 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:50:19.707851 16819 webserver.cc:460] Webserver started at http://127.0.98.62:42435/ using document root <none> and password file <none>
I20260924 04:50:19.708401 16819 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:19.708467 16819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:19.712522 16819 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:50:19.715004 16834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:19.716400 16819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:50:19.716532 16819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "ebcdcf67db39499293d8a608a5cb4a79"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "ade310c1610f57b6549f70ad1bff6cd3"
server_key_iv: "00ddd90136f7b50cb3732425f1320504"
server_key_version: "encryptionkey@0"
I20260924 04:50:19.717108 16819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:19.727435 16819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:19.730625 16819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:19.730820 16819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:19.737739 16819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34241
I20260924 04:50:19.738018 16886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34241 every 8 connection(s)
I20260924 04:50:19.738829 16819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:19.741607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 16819
I20260924 04:50:19.742106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42927
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34241
--webserver_port=36721
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:50:19.743752 16887 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:50:19.745481 16887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Bootstrap starting.
I20260924 04:50:19.762843 16887 log.cc:824] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:19.774288 16887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:50:19.774685 16887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Bootstrap complete.
I20260924 04:50:19.778712 16887 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 04:50:19.779191 16887 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcdcf67db39499293d8a608a5cb4a79, State: Initialized, Role: FOLLOWER
I20260924 04:50:19.779736 16887 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 04:50:19.779862 16887 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:19.779929 16887 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:19.780023 16887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:50:19.782043 16887 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 04:50:19.782377 16887 leader_election.cc:302] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebcdcf67db39499293d8a608a5cb4a79; no voters:
I20260924 04:50:19.782689 16887 leader_election.cc:288] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 04:50:19.782840 16891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:50:19.783119 16891 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [term 3 LEADER]: Becoming Leader. State: Replica: ebcdcf67db39499293d8a608a5cb4a79, State: Running, Role: LEADER
I20260924 04:50:19.783491 16891 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 04:50:19.783952 16887 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:19.785022 16893 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebcdcf67db39499293d8a608a5cb4a79. Latest consensus state: current_term: 3 leader_uuid: "ebcdcf67db39499293d8a608a5cb4a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } } }
I20260924 04:50:19.785151 16893 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:19.785444 16892 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ebcdcf67db39499293d8a608a5cb4a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcdcf67db39499293d8a608a5cb4a79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } } }
I20260924 04:50:19.785521 16892 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:19.785960 16900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:19.791571 16900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:19.792250 16900 catalog_manager.cc:1259] Loaded cluster ID: f32e0c94c01e4ae49615adfb3984a307
I20260924 04:50:19.792317 16900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:19.795166 16900 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:19.795928 16900 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Loaded TSK: 0
I20260924 04:50:19.796020 16900 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ebcdcf67db39499293d8a608a5cb4a79: Loaded TSK: 1
F20260924 04:50:19.796748 16900 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2cb0ade9ad google::LogMessage::Fail() at ??:0
@ 0x7f2cb0ae188a google::LogMessage::SendToLog() at ??:0
@ 0x7f2cb0ade4de google::LogMessage::Flush() at ??:0
@ 0x7f2cb0adfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2cb37a218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2cb379af0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2cbb4437bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2cbb41d99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2cbb41e178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f2cbb45fdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f2cbb47e82b std::function<>::operator()() at ??:0
@ 0x7f2cbb41db4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f2cbb41ef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2cbb41b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f2cbb45f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2cb2d303d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2cb2d30c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f2cb2d323f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2cb2d21512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2cb43656db start_thread at ??:0
@ 0x7f2caf5dc61f clone at ??:0
W20260924 04:50:19.888610 16889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:19.889009 16889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:19.889081 16889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:19.893957 16889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:19.894033 16889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:19.894100 16889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:50:19.899950 16889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36721
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:19.901057 16889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:19.902093 16889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:19.911490 16935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:19.911599 16937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:19.911492 16934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:19.911868 16889 server_base.cc:1034] running on GCE node
I20260924 04:50:19.912251 16889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:19.912909 16889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:19.914083 16889 hybrid_clock.cc:648] HybridClock initialized: now 1790225419914072 us; error 28 us; skew 500 ppm
Sep 24 04:50:19 dist-test-slave-hvcl krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:19.916844 16889 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:50:19.917742 16889 webserver.cc:460] Webserver started at http://127.0.98.1:36721/ using document root <none> and password file <none>
I20260924 04:50:19.918284 16889 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:19.918350 16889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:19.922365 16889 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:50:19.926180 16944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:19.927606 16889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:50:19.927729 16889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "9b3070d2261747659b8c90867bd9ae36"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "043bca0ce576c43476cdab4a3ce52d37"
server_key_iv: "809d7e5fbaa6be9b4446099daceecaca"
server_key_version: "encryptionkey@0"
I20260924 04:50:19.928372 16889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:19.949098 16889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:19.952299 16889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:19.952476 16889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:19.953102 16889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:19.954281 16951 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:50:19.958352 16889 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:50:19.958411 16889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260924 04:50:19.958469 16889 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:50:19.960186 16889 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:50:19.960242 16889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260924 04:50:19.960542 16951 tablet_bootstrap.cc:492] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36: Bootstrap starting.
I20260924 04:50:19.970299 16889 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42927
I20260924 04:50:19.970636 17058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42927 every 8 connection(s)
I20260924 04:50:19.971653 16889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:50:19.979578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 16889
I20260924 04:50:19.980078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42657
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34241
--webserver_port=41753
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:50:19.980741 16951 log.cc:824] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:19.990386 16951 tablet_bootstrap.cc:492] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:50:19.990818 16951 tablet_bootstrap.cc:492] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36: Bootstrap complete.
I20260924 04:50:19.992178 16951 ts_tablet_manager.cc:1397] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
I20260924 04:50:19.995684 16951 raft_consensus.cc:348] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:50:19.996275 16951 raft_consensus.cc:729] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3070d2261747659b8c90867bd9ae36, State: Initialized, Role: FOLLOWER
I20260924 04:50:19.996845 16951 consensus_queue.cc:260] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:50:19.998044 16951 ts_tablet_manager.cc:1428] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 04:50:20.022657 17061 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:50:19.974506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:38753)
0924 04:50:19.975534 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:19.975600 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:19.976730 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:20.021889 (+ 45159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":262,"threads_started":1}
W20260924 04:50:20.023249 17059 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34241 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34241: Client connection negotiation failed: client connection to 127.0.98.62:34241: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:50:20.024055 17061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:20.023752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:36355)
0924 04:50:20.023928 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:20.023996 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 04:50:20.024894 17061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:20.024600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:40903)
0924 04:50:20.024771 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:20.024832 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 04:50:20.025099 17059 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:50:20.099334 17063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:20.099725 17063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:20.099790 17063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:20.103978 17063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:20.104043 17063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:20.104107 17063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:50:20.110914 17063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41753
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:20.112083 17063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:20.113096 17063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:20.122895 17072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:20.122949 17070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:20.123145 17063 server_base.cc:1034] running on GCE node
W20260924 04:50:20.123176 17069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:20.123718 17063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:20.124410 17063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:20.125583 17063 hybrid_clock.cc:648] HybridClock initialized: now 1790225420125563 us; error 33 us; skew 500 ppm
Sep 24 04:50:20 dist-test-slave-hvcl krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:20.128233 17063 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:50:20.129136 17063 webserver.cc:460] Webserver started at http://127.0.98.2:41753/ using document root <none> and password file <none>
I20260924 04:50:20.129688 17063 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:20.129755 17063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:20.133793 17063 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:50:20.136123 17079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:20.137487 17063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:50:20.137611 17063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db"
format_stamp: "Formatted at 2026-09-24 04:49:37 on dist-test-slave-hvcl"
server_key: "ccc980eb97cacd6345412a992b1b028f"
server_key_iv: "eda93f12a8b122c18bc8365cb12a4118"
server_key_version: "encryptionkey@0"
I20260924 04:50:20.138192 17063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:20.153160 17063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:20.156366 17063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:20.156543 17063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:20.157114 17063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:20.157908 17063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:20.157956 17063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:20.158010 17063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:20.158035 17063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:20.166779 17063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42657
I20260924 04:50:20.166826 17192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42657 every 8 connection(s)
I20260924 04:50:20.167855 17063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:50:20.171838 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:20.170163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:52973)
0924 04:50:20.170897 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:20.171084 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":113,"threads_started":1}
W20260924 04:50:20.172346 17193 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34241 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34241: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
I20260924 04:50:20.172909 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:20.172689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:54671)
0924 04:50:20.172799 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:20.172850 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 04:50:20.173447 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:20.173270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:56285)
0924 04:50:20.173359 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:20.173398 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 04:50:20.173590 17193 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:50:20.177064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 17063
I20260924 04:50:20.177450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42755
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34241
--webserver_port=45287
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:20.288273 17196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:20.288559 17196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:20.288602 17196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:20.292766 17196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:20.292841 17196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:20.292904 17196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:50:20.298328 17196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45287
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:20.299484 17196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:20.300509 17196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:20.310251 17204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:20.310321 17201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:20.310433 17202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:20.310534 17196 server_base.cc:1034] running on GCE node
I20260924 04:50:20.311065 17196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:20.311740 17196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:20.312970 17196 hybrid_clock.cc:648] HybridClock initialized: now 1790225420312931 us; error 54 us; skew 500 ppm
Sep 24 04:50:20 dist-test-slave-hvcl krb5kdc[15743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:20.315845 17196 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:50:20.316743 17196 webserver.cc:460] Webserver started at http://127.0.98.3:45287/ using document root <none> and password file <none>
I20260924 04:50:20.317277 17196 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:20.317334 17196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:20.321189 17196 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:50:20.323217 17211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:20.324517 17196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:50:20.324621 17196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "e20568f2f5e74956bc62367868e4c8a4"
format_stamp: "Formatted at 2026-09-24 04:49:38 on dist-test-slave-hvcl"
server_key: "a2e28863cc31bb4ebb9f92e78de14743"
server_key_iv: "61ed705fcfaf6e88a217df84889830cc"
server_key_version: "encryptionkey@0"
I20260924 04:50:20.325160 17196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:20.342669 17196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:20.346030 17196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:20.346189 17196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:20.346740 17196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:20.347584 17196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:20.347633 17196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:20.347685 17196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:20.347714 17196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:20.356099 17196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42755
I20260924 04:50:20.356179 17324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42755 every 8 connection(s)
I20260924 04:50:20.357124 17196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:50:20.361061 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:20.359315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:37529)
0924 04:50:20.359813 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:20.360320 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":114,"threads_started":1}
W20260924 04:50:20.361509 17325 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34241 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34241: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
I20260924 04:50:20.362146 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:20.361929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:53317)
0924 04:50:20.362040 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:20.362089 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 04:50:20.362663 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:20.362489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:38463)
0924 04:50:20.362575 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:20.362614 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 04:50:20.362807 17325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:50:20.363909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 17196
I20260924 04:50:21.026167 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:21.025694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:38937)
0924 04:50:21.025993 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:21.026056 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":122,"threads_started":1}
I20260924 04:50:21.174705 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:21.174258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:48821)
0924 04:50:21.174533 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:21.174602 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":91,"threads_started":1}
I20260924 04:50:21.363752 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:21.363370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:34631)
0924 04:50:21.363602 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:21.363650 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
I20260924 04:50:21.488572 17332 raft_consensus.cc:482] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:50:21.488782 17332 raft_consensus.cc:504] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:50:21.489907 17332 leader_election.cc:288] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657), e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755)
Sep 24 04:50:21 dist-test-slave-hvcl krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 04:50:21 dist-test-slave-hvcl krb5kdc[15743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 04:50:21.503397 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:21.490197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42755 (local address 127.0.98.1:41143)
0924 04:50:21.490530 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:21.490542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:21.490649 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:21.492125 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:21.492132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:21.492652 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:21.493132 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:21.493147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:21.494447 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:21.494451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:21.495498 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:21.495508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:21.496174 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:21.496178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:21.496386 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:21.497063 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:50:21.497075 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:21.498593 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:21.500828 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:21.500835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:21.500849 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:21.501284 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:21.501624 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:21.501627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:21.501628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:21.501785 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:21.502442 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:21.502447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:21.502664 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 04:50:21.502829 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 04:50:21.503180 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":113,"spinlock_wait_cycles":42112,"thread_start_us":70,"threads_started":1}
I20260924 04:50:21.503518 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:21.490106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42657 (local address 127.0.98.1:55701)
0924 04:50:21.490269 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:21.490284 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:21.490386 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:21.492125 (+ 1739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:21.492132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:21.492650 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:21.493132 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:21.493147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:21.494378 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:21.494382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:21.495496 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:21.495506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:21.496175 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:21.496178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:21.496429 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:21.497024 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:50:21.497040 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:21.498952 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:21.500845 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:21.500848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:21.500851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:21.501273 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:21.501599 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:21.501602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:21.501604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:21.501785 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:21.502960 (+ 1175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:21.502963 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:21.503083 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 04:50:21.503171 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 04:50:21.503316 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":201,"spinlock_wait_cycles":19328}
I20260924 04:50:21.503677 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:21.490341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41143 (local address 127.0.98.3:42755)
0924 04:50:21.490758 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:21.490776 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:21.490805 (+ 29us) server_negotiation.cc:407] Connection header received
0924 04:50:21.491499 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:21.491504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:21.491798 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:21.492061 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:21.493295 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:21.494310 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:21.495605 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:21.496058 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:21.496205 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:21.498814 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:21.498821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:21.498839 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:21.498874 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:21.500703 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:21.501396 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:21.501400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:21.501404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:21.501513 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:21.501876 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:21.501879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:21.501880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:21.502342 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:21.502457 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:21.503044 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 04:50:21.503321 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":101,"threads_started":1}
I20260924 04:50:21.503787 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:21.490310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55701 (local address 127.0.98.2:42657)
0924 04:50:21.490626 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:21.490647 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:21.490682 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:50:21.491403 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:21.491410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:21.491771 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:21.492049 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:21.493296 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:21.494254 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:21.495604 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:21.496059 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:21.496209 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:21.499140 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:21.499145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:21.499156 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:21.499185 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:21.500724 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:21.501378 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:21.501385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:21.501396 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:21.501511 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:21.502498 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:21.502500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:21.502501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:21.502853 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:21.503002 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:21.503330 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 04:50:21.503620 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":96,"threads_started":1}
W20260924 04:50:21.506251 16946 leader_election.cc:341] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657): Not found: Tablet not found: edc3f7c89410426bab8d751c9028b56d
W20260924 04:50:21.506404 16947 leader_election.cc:341] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755): Not found: Tablet not found: edc3f7c89410426bab8d751c9028b56d
I20260924 04:50:21.506474 16947 leader_election.cc:302] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b3070d2261747659b8c90867bd9ae36; no voters: 5f36a7e4b4ca463ab97ad261f3a5f0db, e20568f2f5e74956bc62367868e4c8a4
I20260924 04:50:21.506697 17332 raft_consensus.cc:2726] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:50:22.027482 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:22.027037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:34335)
0924 04:50:22.027312 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:22.027379 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
I20260924 04:50:22.175855 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:22.175444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:59269)
0924 04:50:22.175700 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:22.175760 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
I20260924 04:50:22.364878 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:22.364468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:36803)
0924 04:50:22.364706 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:22.364770 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260924 04:50:23.028690 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:23.028263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:51095)
0924 04:50:23.028531 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:23.028595 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260924 04:50:23.109139 17340 raft_consensus.cc:482] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:50:23.109259 17340 raft_consensus.cc:504] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:50:23.109624 17340 leader_election.cc:288] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657), e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755)
W20260924 04:50:23.110414 16947 leader_election.cc:341] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755): Not found: Tablet not found: edc3f7c89410426bab8d751c9028b56d
W20260924 04:50:23.110538 16946 leader_election.cc:341] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657): Not found: Tablet not found: edc3f7c89410426bab8d751c9028b56d
I20260924 04:50:23.110634 16946 leader_election.cc:302] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b3070d2261747659b8c90867bd9ae36; no voters: 5f36a7e4b4ca463ab97ad261f3a5f0db, e20568f2f5e74956bc62367868e4c8a4
I20260924 04:50:23.110778 17340 raft_consensus.cc:2726] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:50:23.177030 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:23.176574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:36355)
0924 04:50:23.176832 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:23.176919 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
I20260924 04:50:23.366079 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:23.365650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:46911)
0924 04:50:23.365909 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:23.365971 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260924 04:50:24.029904 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:24.029480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:45225)
0924 04:50:24.029736 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:24.029800 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
I20260924 04:50:24.178323 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:24.177867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:58767)
0924 04:50:24.178155 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:24.178222 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":78,"threads_started":1}
I20260924 04:50:24.367302 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:24.366838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:33077)
0924 04:50:24.367140 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:24.367205 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":82,"threads_started":1}
I20260924 04:50:25.031193 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:25.030757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:57173)
0924 04:50:25.031037 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:25.031097 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260924 04:50:25.065940 17347 raft_consensus.cc:482] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:50:25.066072 17347 raft_consensus.cc:504] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:50:25.066478 17347 leader_election.cc:288] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657), e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755)
W20260924 04:50:25.067261 16946 leader_election.cc:341] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657): Not found: Tablet not found: edc3f7c89410426bab8d751c9028b56d
W20260924 04:50:25.067375 16947 leader_election.cc:341] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755): Not found: Tablet not found: edc3f7c89410426bab8d751c9028b56d
I20260924 04:50:25.067471 16947 leader_election.cc:302] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b3070d2261747659b8c90867bd9ae36; no voters: 5f36a7e4b4ca463ab97ad261f3a5f0db, e20568f2f5e74956bc62367868e4c8a4
I20260924 04:50:25.067613 17347 raft_consensus.cc:2726] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:50:25.179693 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:25.179203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:53241)
0924 04:50:25.179512 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:25.179578 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":72,"threads_started":1}
I20260924 04:50:25.368381 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:25.368008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:39763)
0924 04:50:25.368229 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:25.368281 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
I20260924 04:50:26.032451 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:26.031995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:49521)
0924 04:50:26.032282 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:26.032351 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
I20260924 04:50:26.180871 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:26.180437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:34285)
0924 04:50:26.180715 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:26.180775 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260924 04:50:26.369498 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:26.369079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:58361)
0924 04:50:26.369348 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:26.369405 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":69,"threads_started":1}
I20260924 04:50:27.033612 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:27.033193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:48803)
0924 04:50:27.033443 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:27.033508 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260924 04:50:27.182109 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:27.181587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:53799)
0924 04:50:27.181959 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:27.182014 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":66,"threads_started":1}
I20260924 04:50:27.370667 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:27.370222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:45199)
0924 04:50:27.370516 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:27.370572 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":70,"threads_started":1}
I20260924 04:50:27.693794 17356 raft_consensus.cc:482] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:50:27.693935 17356 raft_consensus.cc:504] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:50:27.694370 17356 leader_election.cc:288] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657), e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755)
W20260924 04:50:27.695142 16947 leader_election.cc:341] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755): Not found: Tablet not found: edc3f7c89410426bab8d751c9028b56d
W20260924 04:50:27.695241 16946 leader_election.cc:341] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657): Not found: Tablet not found: edc3f7c89410426bab8d751c9028b56d
I20260924 04:50:27.695405 16946 leader_election.cc:302] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b3070d2261747659b8c90867bd9ae36; no voters: 5f36a7e4b4ca463ab97ad261f3a5f0db, e20568f2f5e74956bc62367868e4c8a4
I20260924 04:50:27.695554 17356 raft_consensus.cc:2726] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:50:28.034888 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:28.034442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:42439)
0924 04:50:28.034714 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:28.034781 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20260924 04:50:28.183324 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:28.182873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:52071)
0924 04:50:28.183166 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:28.183230 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":107,"threads_started":1}
I20260924 04:50:28.371840 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:28.371379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:40933)
0924 04:50:28.371616 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:28.371672 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
I20260924 04:50:29.036162 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:29.035656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:55001)
0924 04:50:29.036005 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:29.036067 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":152,"threads_started":1}
I20260924 04:50:29.184630 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:29.184168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:36033)
0924 04:50:29.184471 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:29.184529 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
I20260924 04:50:29.373047 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:29.372615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:49621)
0924 04:50:29.372876 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:29.372931 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260924 04:50:30.037496 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:30.037061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:39193)
0924 04:50:30.037332 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:30.037398 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260924 04:50:30.185842 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:30.185422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:60497)
0924 04:50:30.185669 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:30.185730 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260924 04:50:30.374222 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:30.373806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:38577)
0924 04:50:30.374064 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:30.374122 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
I20260924 04:50:31.038653 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:31.038203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:46525)
0924 04:50:31.038489 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:31.038556 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260924 04:50:31.187091 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:31.186560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:54743)
0924 04:50:31.186900 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:31.186958 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":88,"threads_started":1}
I20260924 04:50:31.375464 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:31.375031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:54225)
0924 04:50:31.375308 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:31.375367 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260924 04:50:32.039955 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:32.039515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:33629)
0924 04:50:32.039787 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:32.039857 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260924 04:50:32.188163 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:32.187755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:36779)
0924 04:50:32.188016 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:32.188071 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
I20260924 04:50:32.376725 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:32.376264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:38461)
0924 04:50:32.376558 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:32.376626 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":85,"threads_started":1}
I20260924 04:50:33.041301 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:33.040837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:47695)
0924 04:50:33.041128 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:33.041198 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
I20260924 04:50:33.189327 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:33.188926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:37669)
0924 04:50:33.189167 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:33.189231 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20260924 04:50:33.378047 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:33.377568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:42121)
0924 04:50:33.377886 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:33.377948 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
I20260924 04:50:34.042608 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:34.042182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:59647)
0924 04:50:34.042445 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:34.042512 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260924 04:50:34.190618 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:34.190183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:52149)
0924 04:50:34.190456 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:34.190523 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":96,"threads_started":1}
I20260924 04:50:34.379407 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:34.378904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:37795)
0924 04:50:34.379237 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:34.379310 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":138,"threads_started":1}
I20260924 04:50:34.718114 17378 raft_consensus.cc:482] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:50:34.718273 17378 raft_consensus.cc:504] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f36a7e4b4ca463ab97ad261f3a5f0db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42657 } } peers { permanent_uuid: "e20568f2f5e74956bc62367868e4c8a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42755 } } peers { permanent_uuid: "9b3070d2261747659b8c90867bd9ae36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42927 } }
I20260924 04:50:34.718698 17378 leader_election.cc:288] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657), e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755)
W20260924 04:50:34.719515 16946 leader_election.cc:341] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f36a7e4b4ca463ab97ad261f3a5f0db (127.0.98.2:42657): Not found: Tablet not found: edc3f7c89410426bab8d751c9028b56d
W20260924 04:50:34.719620 16947 leader_election.cc:341] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e20568f2f5e74956bc62367868e4c8a4 (127.0.98.3:42755): Not found: Tablet not found: edc3f7c89410426bab8d751c9028b56d
I20260924 04:50:34.719722 16947 leader_election.cc:302] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b3070d2261747659b8c90867bd9ae36; no voters: 5f36a7e4b4ca463ab97ad261f3a5f0db, e20568f2f5e74956bc62367868e4c8a4
I20260924 04:50:34.719864 17378 raft_consensus.cc:2726] T edc3f7c89410426bab8d751c9028b56d P 9b3070d2261747659b8c90867bd9ae36 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:50:35.044126 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:35.043617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:38759)
0924 04:50:35.043961 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:35.044027 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":133,"threads_started":1}
I20260924 04:50:35.192212 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:35.191636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:59133)
0924 04:50:35.192005 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:35.192081 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":176,"threads_started":1}
I20260924 04:50:35.367795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 16889
I20260924 04:50:35.373140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 17063
I20260924 04:50:35.377882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 17196
2026-09-24T04:50:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59139 ms)
[----------] 30 tests from SecurityITest (240692 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:50:35 dist-test-slave-hvcl krb5kdc[17383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:50:35 dist-test-slave-hvcl krb5kdc[17383](info): set up 2 sockets
Sep 24 04:50:35 dist-test-slave-hvcl krb5kdc[17383](info): commencing operation
krb5kdc: starting...
W20260924 04:50:36.350811 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:50:36 dist-test-slave-hvcl krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:50:36.471663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34591
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:36.579833 17396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:36.580114 17396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:36.580155 17396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:36.584115 17396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:36.584172 17396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:36.584190 17396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:36.584205 17396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:36.584219 17396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:36.589450 17396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:36.590463 17396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:36.591446 17396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:36.600562 17402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:36.600562 17404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:36.600567 17401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:36.600605 17396 server_base.cc:1034] running on GCE node
I20260924 04:50:36.601454 17396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:36.602626 17396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:36.603801 17396 hybrid_clock.cc:648] HybridClock initialized: now 1790225436603781 us; error 34 us; skew 500 ppm
Sep 24 04:50:36 dist-test-slave-hvcl krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:36.606609 17396 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:50:36.607604 17396 webserver.cc:460] Webserver started at http://127.0.98.62:42835/ using document root <none> and password file <none>
I20260924 04:50:36.608191 17396 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:36.608258 17396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:36.608439 17396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:36.610810 17396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "e6f2a39f286e43b1a9258f5003e82132"
format_stamp: "Formatted at 2026-09-24 04:50:36 on dist-test-slave-hvcl"
server_key: "8e71fcc9e494d6acad464b1fdccb88b0"
server_key_iv: "6ca6b5379949bce732589376589fd848"
server_key_version: "encryptionkey@0"
I20260924 04:50:36.611395 17396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "e6f2a39f286e43b1a9258f5003e82132"
format_stamp: "Formatted at 2026-09-24 04:50:36 on dist-test-slave-hvcl"
server_key: "8e71fcc9e494d6acad464b1fdccb88b0"
server_key_iv: "6ca6b5379949bce732589376589fd848"
server_key_version: "encryptionkey@0"
I20260924 04:50:36.615211 17396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:50:36.617693 17411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:36.618925 17396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:50:36.619068 17396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "e6f2a39f286e43b1a9258f5003e82132"
format_stamp: "Formatted at 2026-09-24 04:50:36 on dist-test-slave-hvcl"
server_key: "8e71fcc9e494d6acad464b1fdccb88b0"
server_key_iv: "6ca6b5379949bce732589376589fd848"
server_key_version: "encryptionkey@0"
I20260924 04:50:36.619174 17396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:36.633993 17396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:36.637162 17396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:36.637339 17396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:36.643683 17396 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34591
I20260924 04:50:36.643721 17463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34591 every 8 connection(s)
I20260924 04:50:36.644587 17396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:36.647225 17464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:36.647648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 17396
I20260924 04:50:36.647758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:36.648026 392 external_mini_cluster.cc:1428] Setting key a45bd6e3cebefc86876c6135f6e1a29a
I20260924 04:50:36.654203 17464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132: Bootstrap starting.
I20260924 04:50:36.656774 17464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132: Neither blocks nor log segments found. Creating new log.
Sep 24 04:50:36 dist-test-slave-hvcl krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:50:36.657514 17464 log.cc:824] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:36.659803 17464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132: No bootstrap required, opened a new log
I20260924 04:50:36.662703 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:36.649333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58304 (local address 127.0.98.62:34591)
0924 04:50:36.649744 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:36.649762 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:36.649806 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:50:36.650529 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:36.650555 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:36.651031 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:36.651409 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:36.652058 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:36.653542 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:36.654446 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:36.655262 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:36.655434 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:36.657872 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:36.657879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:36.657897 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:36.657927 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:36.659782 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:36.660207 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:36.660212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:36.660218 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:36.660374 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:36.660595 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:36.660598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:36.660600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:36.661074 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:36.661264 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:36.661433 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 04:50:36.661731 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20260924 04:50:36.664180 17464 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f2a39f286e43b1a9258f5003e82132" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34591 } }
I20260924 04:50:36.664393 17464 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:36.664425 17464 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f2a39f286e43b1a9258f5003e82132, State: Initialized, Role: FOLLOWER
I20260924 04:50:36.664887 17464 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f2a39f286e43b1a9258f5003e82132" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34591 } }
I20260924 04:50:36.665045 17464 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:36.665110 17464 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:36.665201 17464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:36.666057 17464 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f2a39f286e43b1a9258f5003e82132" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34591 } }
I20260924 04:50:36.666364 17464 leader_election.cc:302] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6f2a39f286e43b1a9258f5003e82132; no voters:
I20260924 04:50:36.666640 17464 leader_election.cc:288] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:36.666749 17469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:36.666940 17469 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [term 1 LEADER]: Becoming Leader. State: Replica: e6f2a39f286e43b1a9258f5003e82132, State: Running, Role: LEADER
I20260924 04:50:36.667297 17469 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f2a39f286e43b1a9258f5003e82132" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34591 } }
I20260924 04:50:36.667905 17464 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:36.668829 17471 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6f2a39f286e43b1a9258f5003e82132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f2a39f286e43b1a9258f5003e82132" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34591 } } }
I20260924 04:50:36.668959 17471 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:36.669317 17478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:36.669865 17470 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6f2a39f286e43b1a9258f5003e82132. Latest consensus state: current_term: 1 leader_uuid: "e6f2a39f286e43b1a9258f5003e82132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f2a39f286e43b1a9258f5003e82132" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34591 } } }
I20260924 04:50:36.670097 17470 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:36.672580 17478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:36.678153 17478 catalog_manager.cc:1347] Generated new cluster ID: 9e29723a21844856b342e2a92ec4f41d
I20260924 04:50:36.678233 17478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:36.689868 17478 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:36.690960 17478 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:36.705205 17478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132: Generated new TSK 0
I20260924 04:50:36.706139 17478 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:50:36.738415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34591
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:36.850787 17490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:36.851109 17490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:36.851162 17490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:36.855252 17490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:36.855319 17490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:36.855378 17490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:50:36.860692 17490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:36.861861 17490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:36.862907 17490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:36.872632 17495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:36.872779 17490 server_base.cc:1034] running on GCE node
W20260924 04:50:36.872653 17498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:36.872644 17496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:36.873318 17490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:36.874006 17490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:36.875168 17490 hybrid_clock.cc:648] HybridClock initialized: now 1790225436875144 us; error 40 us; skew 500 ppm
Sep 24 04:50:36 dist-test-slave-hvcl krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:36.878078 17490 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:50:36.879168 17490 webserver.cc:460] Webserver started at http://127.0.98.1:43069/ using document root <none> and password file <none>
I20260924 04:50:36.879737 17490 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:36.879803 17490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:36.880012 17490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:36.882061 17490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5"
format_stamp: "Formatted at 2026-09-24 04:50:36 on dist-test-slave-hvcl"
server_key: "3b33e3fe52d0630414e43f3e94d53b67"
server_key_iv: "ee11af392929947f97b52611d732f37c"
server_key_version: "encryptionkey@0"
I20260924 04:50:36.882598 17490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5"
format_stamp: "Formatted at 2026-09-24 04:50:36 on dist-test-slave-hvcl"
server_key: "3b33e3fe52d0630414e43f3e94d53b67"
server_key_iv: "ee11af392929947f97b52611d732f37c"
server_key_version: "encryptionkey@0"
I20260924 04:50:36.886248 17490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:50:36.888746 17505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:36.890012 17490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:50:36.890113 17490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5"
format_stamp: "Formatted at 2026-09-24 04:50:36 on dist-test-slave-hvcl"
server_key: "3b33e3fe52d0630414e43f3e94d53b67"
server_key_iv: "ee11af392929947f97b52611d732f37c"
server_key_version: "encryptionkey@0"
I20260924 04:50:36.890220 17490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:36.911302 17490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:36.914625 17490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:36.914803 17490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:36.915449 17490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:36.916273 17490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:36.916318 17490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:36.916402 17490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:36.916435 17490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:36.924932 17490 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35037
I20260924 04:50:36.924983 17618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35037 every 8 connection(s)
I20260924 04:50:36.925916 17490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:50:36 dist-test-slave-hvcl krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:50:36.935240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 17490
I20260924 04:50:36.935357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:50:36.935662 392 external_mini_cluster.cc:1428] Setting key 1119c9d478fa492e3ece1514beff114d
I20260924 04:50:36.939338 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:36.927888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50667 (local address 127.0.98.62:34591)
0924 04:50:36.928034 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:36.928039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:36.928901 (+ 862us) server_negotiation.cc:407] Connection header received
0924 04:50:36.929760 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:36.929762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:36.929810 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:36.929889 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:36.931091 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:36.931614 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:36.932750 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:36.932981 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:36.933053 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:36.935561 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:36.935564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:36.935567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:36.935598 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:36.936949 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:36.937604 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:36.937608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:36.937609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:36.937699 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:36.938076 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:36.938079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:36.938080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:36.938292 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:36.938390 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:36.939035 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 04:50:36.939175 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:50:36.940263 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:36.928244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34591 (local address 127.0.98.1:50667)
0924 04:50:36.928735 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:36.928782 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:36.929568 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:36.930034 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:36.930041 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:36.930431 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:36.930925 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:36.930939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:36.931749 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:36.931752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:36.932652 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:36.932661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:36.933106 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:36.933109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:36.933279 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:36.933802 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:50:36.933818 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:36.935321 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:36.937067 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:36.937073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:36.937091 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:36.937496 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:36.937820 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:36.937823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:36.937825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:36.937987 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:36.938407 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:36.938413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:36.938706 (+ 293us) client_negotiation.cc:769] Sending connection context
0924 04:50:36.938943 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 04:50:36.939267 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":123,"threads_started":1}
I20260924 04:50:36.941511 17619 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34591
I20260924 04:50:36.941779 17619 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:36.942391 17619 heartbeater.cc:511] Master 127.0.98.62:34591 requested a full tablet report, sending...
I20260924 04:50:36.943953 17428 ts_manager.cc:194] Registered new tserver with Master: 63820ccf6c8740ed8ed6b3cd48b6afc5 (127.0.98.1:35037)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:50:36.945098 17428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:50:36.962785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34591
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:37.073004 17624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:37.073282 17624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:37.073328 17624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:37.077524 17624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:37.077608 17624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:37.077687 17624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:50:37.083335 17624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:37.084479 17624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:37.085515 17624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:37.095191 17632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:37.095245 17624 server_base.cc:1034] running on GCE node
W20260924 04:50:37.095448 17629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:37.095252 17630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:37.095956 17624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:37.096609 17624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:37.097769 17624 hybrid_clock.cc:648] HybridClock initialized: now 1790225437097732 us; error 48 us; skew 500 ppm
Sep 24 04:50:37 dist-test-slave-hvcl krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:37.100505 17624 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:50:37.101414 17624 webserver.cc:460] Webserver started at http://127.0.98.2:41973/ using document root <none> and password file <none>
I20260924 04:50:37.101960 17624 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:37.102016 17624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:37.102180 17624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:37.104357 17624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "58349eb542324ec9bdf95ad7930333d6"
format_stamp: "Formatted at 2026-09-24 04:50:37 on dist-test-slave-hvcl"
server_key: "e4a8c3c129eb51ea56e9ade691ba9d37"
server_key_iv: "10062d2546c5764cc9588c4f85375d3d"
server_key_version: "encryptionkey@0"
I20260924 04:50:37.104911 17624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "58349eb542324ec9bdf95ad7930333d6"
format_stamp: "Formatted at 2026-09-24 04:50:37 on dist-test-slave-hvcl"
server_key: "e4a8c3c129eb51ea56e9ade691ba9d37"
server_key_iv: "10062d2546c5764cc9588c4f85375d3d"
server_key_version: "encryptionkey@0"
I20260924 04:50:37.108984 17624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:50:37.111604 17639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:37.112962 17624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:50:37.113085 17624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "58349eb542324ec9bdf95ad7930333d6"
format_stamp: "Formatted at 2026-09-24 04:50:37 on dist-test-slave-hvcl"
server_key: "e4a8c3c129eb51ea56e9ade691ba9d37"
server_key_iv: "10062d2546c5764cc9588c4f85375d3d"
server_key_version: "encryptionkey@0"
I20260924 04:50:37.113200 17624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:37.130756 17624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:37.134228 17624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:37.134423 17624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:37.135094 17624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:37.135954 17624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:37.136000 17624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:37.136045 17624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:37.136075 17624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:37.144395 17624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36409
I20260924 04:50:37.144416 17752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36409 every 8 connection(s)
I20260924 04:50:37.145277 17624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:50:37.149015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 17624
I20260924 04:50:37.149154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:50:37.149466 392 external_mini_cluster.cc:1428] Setting key ce82e9eb03c17bc07cc387ccbb90b71d
Sep 24 04:50:37 dist-test-slave-hvcl krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:50:37.159488 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.147167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39307 (local address 127.0.98.62:34591)
0924 04:50:37.147300 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:37.147305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:37.148191 (+ 886us) server_negotiation.cc:407] Connection header received
0924 04:50:37.149287 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:37.149291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:37.149346 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:37.149418 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:37.150745 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.151371 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.152852 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.153071 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.153156 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.155722 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:37.155727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:37.155730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:37.155760 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:37.157309 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:37.157866 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:37.157870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:37.157872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:37.157959 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:37.158327 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:37.158330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:37.158332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:37.158576 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:37.158680 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:37.159247 (+ 567us) server_negotiation.cc:299] Negotiation successful
0924 04:50:37.159365 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:50:37.160375 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.147572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34591 (local address 127.0.98.2:39307)
0924 04:50:37.148035 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:37.148080 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:37.148967 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:37.149619 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:37.149629 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:37.150066 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:37.150593 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:37.150609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.151489 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.151492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:37.152737 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:37.152747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.153201 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.153205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:37.153365 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.153910 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:50:37.153930 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:37.155543 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:37.157443 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:37.157449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:37.157464 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:37.157761 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:37.158080 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:37.158083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:37.158084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:37.158231 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:37.158678 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:37.158683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:37.158951 (+ 268us) client_negotiation.cc:769] Sending connection context
0924 04:50:37.159173 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 04:50:37.159462 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:50:37.161635 17753 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34591
I20260924 04:50:37.161912 17753 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:37.162497 17753 heartbeater.cc:511] Master 127.0.98.62:34591 requested a full tablet report, sending...
I20260924 04:50:37.163748 17428 ts_manager.cc:194] Registered new tserver with Master: 58349eb542324ec9bdf95ad7930333d6 (127.0.98.2:36409)
I20260924 04:50:37.164567 17428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:50:37.178810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34591
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:37.287981 17758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:37.288256 17758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:37.288316 17758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:37.292447 17758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:37.292515 17758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:37.292595 17758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:50:37.298072 17758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:37.299203 17758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:37.300223 17758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:37.310004 17763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:37.309957 17766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:37.310063 17758 server_base.cc:1034] running on GCE node
W20260924 04:50:37.309983 17764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:37.310643 17758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:37.311370 17758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:37.312582 17758 hybrid_clock.cc:648] HybridClock initialized: now 1790225437312547 us; error 49 us; skew 500 ppm
Sep 24 04:50:37 dist-test-slave-hvcl krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:37.320189 17758 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:50:37.321162 17758 webserver.cc:460] Webserver started at http://127.0.98.3:33391/ using document root <none> and password file <none>
I20260924 04:50:37.321714 17758 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:37.321779 17758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:37.321969 17758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:37.324165 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "1053f166658e4ea58309d5d625a3cacb"
format_stamp: "Formatted at 2026-09-24 04:50:37 on dist-test-slave-hvcl"
server_key: "de5df9719fdd7b4d8521c8a13a934508"
server_key_iv: "294a809ea060584a36698ee7f265970e"
server_key_version: "encryptionkey@0"
I20260924 04:50:37.324719 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1053f166658e4ea58309d5d625a3cacb"
format_stamp: "Formatted at 2026-09-24 04:50:37 on dist-test-slave-hvcl"
server_key: "de5df9719fdd7b4d8521c8a13a934508"
server_key_iv: "294a809ea060584a36698ee7f265970e"
server_key_version: "encryptionkey@0"
I20260924 04:50:37.328481 17758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:50:37.331012 17773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:37.332324 17758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:50:37.332453 17758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "1053f166658e4ea58309d5d625a3cacb"
format_stamp: "Formatted at 2026-09-24 04:50:37 on dist-test-slave-hvcl"
server_key: "de5df9719fdd7b4d8521c8a13a934508"
server_key_iv: "294a809ea060584a36698ee7f265970e"
server_key_version: "encryptionkey@0"
I20260924 04:50:37.332556 17758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:37.356402 17758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:37.359548 17758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:37.359735 17758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:37.360314 17758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:37.361138 17758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:37.361184 17758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:37.361235 17758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:37.361265 17758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:37.369964 17758 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38891
I20260924 04:50:37.370007 17886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38891 every 8 connection(s)
I20260924 04:50:37.370865 17758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:50:37.375268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 17758
I20260924 04:50:37.375401 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:50:37.375722 392 external_mini_cluster.cc:1428] Setting key f477d35bb5f75167af0be28b10b96f22
Sep 24 04:50:37 dist-test-slave-hvcl krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:50:37.384351 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.372652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42277 (local address 127.0.98.62:34591)
0924 04:50:37.372824 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:37.372827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:37.373691 (+ 864us) server_negotiation.cc:407] Connection header received
0924 04:50:37.374554 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:37.374557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:37.374603 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:37.374677 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:37.376241 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.376922 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.378246 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.378478 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.378553 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.381048 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:37.381051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:37.381053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:37.381075 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:37.382450 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:37.382934 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:37.382936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:37.382937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:37.383037 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:37.383386 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:37.383388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:37.383389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:37.383557 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:37.383624 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:37.384123 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 04:50:37.384226 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 04:50:37.385244 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.373083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34591 (local address 127.0.98.3:42277)
0924 04:50:37.373533 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:37.373576 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:37.374356 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:37.374841 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:37.374848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:37.375377 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:37.376090 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:37.376109 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.377048 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.377051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:37.378128 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:37.378138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.378627 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.378630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:37.378807 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.379437 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:50:37.379453 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:37.380893 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:37.382569 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:37.382576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:37.382590 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:37.382853 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:37.383129 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:37.383131 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:37.383133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:37.383298 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:37.383639 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:37.383645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:37.383928 (+ 283us) client_negotiation.cc:769] Sending connection context
0924 04:50:37.384096 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 04:50:37.384400 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260924 04:50:37.386396 17887 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34591
I20260924 04:50:37.386605 17887 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:37.387121 17887 heartbeater.cc:511] Master 127.0.98.62:34591 requested a full tablet report, sending...
I20260924 04:50:37.388248 17428 ts_manager.cc:194] Registered new tserver with Master: 1053f166658e4ea58309d5d625a3cacb (127.0.98.3:38891)
I20260924 04:50:37.388849 17428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42277
I20260924 04:50:37.389468 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:50:37.398402 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.391387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.0.98.62:34591)
0924 04:50:37.391481 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:37.391485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:37.391719 (+ 234us) server_negotiation.cc:407] Connection header received
0924 04:50:37.391823 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:37.391826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:37.391869 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:37.391931 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:37.392577 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.393080 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.393941 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.394152 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.394235 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.395634 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:37.395638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:37.395641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:37.395666 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:37.396863 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:37.397258 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:37.397260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:37.397261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:37.397336 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:37.397634 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:37.397637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:37.397638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:37.397841 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:37.397949 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:37.398172 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 04:50:37.398266 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 04:50:37.401180 17428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:50:37.404279 17428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:50:37.419629 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.413109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35037 (local address 127.0.0.1:37494)
0924 04:50:37.414126 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:37.414138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:37.414232 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:37.415073 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:37.415079 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:37.415102 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:37.415393 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:37.415399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.416505 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.416509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:37.418117 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:37.418127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.419005 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.419009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:37.419135 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.419177 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 04:50:37.419289 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 04:50:37.419438 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"mutex_wait_us":93,"spinlock_wait_cycles":3968,"thread_start_us":105,"threads_started":1}
I20260924 04:50:37.419629 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.413109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38891 (local address 127.0.0.1:59276)
0924 04:50:37.413511 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:37.413569 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:37.413706 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:37.415192 (+ 1486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:37.415194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:37.415208 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:37.415375 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:37.415386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.416987 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.416991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:37.418016 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:37.418027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.419170 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.419174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:37.419371 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.419393 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:50:37.419447 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 04:50:37.419502 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":176,"thread_start_us":106,"threads_started":1}
I20260924 04:50:37.419919 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.414345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.0.98.3:38891)
0924 04:50:37.414730 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:37.414741 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:37.414761 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:50:37.414828 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:37.414833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:37.415052 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:37.415354 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:37.415506 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.416849 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.418155 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.419035 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.419280 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.419448 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:37.419502 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 04:50:37.419603 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":121,"threads_started":1}
I20260924 04:50:37.420308 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.413250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.0.98.1:35037)
0924 04:50:37.413930 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:37.413947 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:37.414321 (+ 374us) server_negotiation.cc:407] Connection header received
0924 04:50:37.414417 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:37.414422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:37.414649 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:37.414809 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:37.415511 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.416346 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.418244 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.418846 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.418952 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.419154 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:37.420044 (+ 890us) server_negotiation.cc:299] Negotiation successful
0924 04:50:37.420153 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":127,"threads_started":1}
I20260924 04:50:37.421331 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.413905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36409 (local address 127.0.0.1:55096)
0924 04:50:37.414579 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:37.414592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:37.414699 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:37.415122 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:37.415125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:37.415144 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:37.415380 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:37.415389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.416893 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.416897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:37.419331 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:37.419340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.420925 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.420929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:37.421022 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.421048 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:50:37.421122 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:50:37.421197 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":238,"thread_start_us":95,"threads_started":1}
I20260924 04:50:37.421432 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.414071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.0.98.2:36409)
0924 04:50:37.414541 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:37.414552 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:37.414677 (+ 125us) server_negotiation.cc:407] Connection header received
0924 04:50:37.414809 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:37.414814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:37.414958 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:37.415132 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:37.415512 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.416726 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.419474 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.420609 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.420716 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.420857 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:37.421183 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 04:50:37.421285 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":108,"threads_started":1}
I20260924 04:50:37.423255 17821 tablet_service.cc:1467] Processing CreateTablet for tablet 638e5aba56ab49d489bd26ce6a5bc64e (DEFAULT_TABLE table=test-table [id=ec51a4a79dd748df993a6cf87ee0e097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:37.423593 17553 tablet_service.cc:1467] Processing CreateTablet for tablet 638e5aba56ab49d489bd26ce6a5bc64e (DEFAULT_TABLE table=test-table [id=ec51a4a79dd748df993a6cf87ee0e097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:37.424387 17821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638e5aba56ab49d489bd26ce6a5bc64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:37.424103 17687 tablet_service.cc:1467] Processing CreateTablet for tablet 638e5aba56ab49d489bd26ce6a5bc64e (DEFAULT_TABLE table=test-table [id=ec51a4a79dd748df993a6cf87ee0e097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:37.424609 17553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638e5aba56ab49d489bd26ce6a5bc64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:37.425004 17687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638e5aba56ab49d489bd26ce6a5bc64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:37.430804 17904 tablet_bootstrap.cc:492] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb: Bootstrap starting.
I20260924 04:50:37.432343 17905 tablet_bootstrap.cc:492] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6: Bootstrap starting.
I20260924 04:50:37.432466 17906 tablet_bootstrap.cc:492] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5: Bootstrap starting.
I20260924 04:50:37.433054 17904 tablet_bootstrap.cc:654] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:37.434075 17904 log.cc:824] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:37.434537 17906 tablet_bootstrap.cc:654] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:37.434537 17905 tablet_bootstrap.cc:654] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:37.435766 17905 log.cc:824] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:37.435766 17906 log.cc:824] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:37.436218 17904 tablet_bootstrap.cc:492] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb: No bootstrap required, opened a new log
I20260924 04:50:37.436532 17904 ts_tablet_manager.cc:1397] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:50:37.437815 17906 tablet_bootstrap.cc:492] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5: No bootstrap required, opened a new log
I20260924 04:50:37.437812 17905 tablet_bootstrap.cc:492] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6: No bootstrap required, opened a new log
I20260924 04:50:37.438002 17906 ts_tablet_manager.cc:1397] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 04:50:37.438002 17905 ts_tablet_manager.cc:1397] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 04:50:37.439540 17904 raft_consensus.cc:348] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35037 } } peers { permanent_uuid: "1053f166658e4ea58309d5d625a3cacb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38891 } } peers { permanent_uuid: "58349eb542324ec9bdf95ad7930333d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36409 } }
I20260924 04:50:37.439798 17904 raft_consensus.cc:374] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:37.439850 17904 raft_consensus.cc:729] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1053f166658e4ea58309d5d625a3cacb, State: Initialized, Role: FOLLOWER
I20260924 04:50:37.440253 17904 consensus_queue.cc:260] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35037 } } peers { permanent_uuid: "1053f166658e4ea58309d5d625a3cacb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38891 } } peers { permanent_uuid: "58349eb542324ec9bdf95ad7930333d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36409 } }
I20260924 04:50:37.441044 17887 heartbeater.cc:503] Master 127.0.98.62:34591 was elected leader, sending a full tablet report...
I20260924 04:50:37.441790 17905 raft_consensus.cc:348] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35037 } } peers { permanent_uuid: "1053f166658e4ea58309d5d625a3cacb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38891 } } peers { permanent_uuid: "58349eb542324ec9bdf95ad7930333d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36409 } }
I20260924 04:50:37.441792 17906 raft_consensus.cc:348] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35037 } } peers { permanent_uuid: "1053f166658e4ea58309d5d625a3cacb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38891 } } peers { permanent_uuid: "58349eb542324ec9bdf95ad7930333d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36409 } }
I20260924 04:50:37.442051 17905 raft_consensus.cc:374] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:37.442065 17906 raft_consensus.cc:374] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:37.442101 17905 raft_consensus.cc:729] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58349eb542324ec9bdf95ad7930333d6, State: Initialized, Role: FOLLOWER
I20260924 04:50:37.442107 17906 raft_consensus.cc:729] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63820ccf6c8740ed8ed6b3cd48b6afc5, State: Initialized, Role: FOLLOWER
I20260924 04:50:37.442476 17904 ts_tablet_manager.cc:1428] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:50:37.442579 17905 consensus_queue.cc:260] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35037 } } peers { permanent_uuid: "1053f166658e4ea58309d5d625a3cacb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38891 } } peers { permanent_uuid: "58349eb542324ec9bdf95ad7930333d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36409 } }
I20260924 04:50:37.442579 17906 consensus_queue.cc:260] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35037 } } peers { permanent_uuid: "1053f166658e4ea58309d5d625a3cacb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38891 } } peers { permanent_uuid: "58349eb542324ec9bdf95ad7930333d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36409 } }
I20260924 04:50:37.443564 17905 ts_tablet_manager.cc:1428] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260924 04:50:37.443564 17906 ts_tablet_manager.cc:1428] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260924 04:50:37.443922 17753 heartbeater.cc:503] Master 127.0.98.62:34591 was elected leader, sending a full tablet report...
I20260924 04:50:37.444478 17619 heartbeater.cc:503] Master 127.0.98.62:34591 was elected leader, sending a full tablet report...
I20260924 04:50:37.484879 17910 raft_consensus.cc:482] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:50:37.485108 17910 raft_consensus.cc:504] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35037 } } peers { permanent_uuid: "1053f166658e4ea58309d5d625a3cacb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38891 } } peers { permanent_uuid: "58349eb542324ec9bdf95ad7930333d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36409 } }
I20260924 04:50:37.486502 17910 leader_election.cc:288] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63820ccf6c8740ed8ed6b3cd48b6afc5 (127.0.98.1:35037), 58349eb542324ec9bdf95ad7930333d6 (127.0.98.2:36409)
I20260924 04:50:37.490854 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.486819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43523 (local address 127.0.98.1:35037)
0924 04:50:37.486932 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:37.486936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:37.487015 (+ 79us) server_negotiation.cc:407] Connection header received
0924 04:50:37.487186 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:37.487188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:37.487228 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:37.487288 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:37.487532 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.488104 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.489828 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.490267 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.490359 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.490419 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:37.490638 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 04:50:37.490715 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:50:37.490861 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.487244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39359 (local address 127.0.98.2:36409)
0924 04:50:37.487373 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:37.487377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:37.487391 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:50:37.487443 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:37.487446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:37.487496 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:37.487563 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:37.487961 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.488646 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.489923 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.490340 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.490451 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.490511 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:37.490671 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 04:50:37.490744 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:50:37.490871 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.486704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35037 (local address 127.0.98.3:43523)
0924 04:50:37.486930 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:37.486945 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:37.487077 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:37.487311 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:37.487313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:37.487329 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:37.487443 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:37.487450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.488775 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.488779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:37.489692 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:37.489701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.490395 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.490399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:37.490502 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.490524 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:50:37.490588 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:50:37.490685 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":51}
I20260924 04:50:37.490876 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.486788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36409 (local address 127.0.98.3:39359)
0924 04:50:37.487123 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:37.487136 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:37.487237 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:37.487675 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:37.487679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:37.487695 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:37.487833 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:37.487841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.488757 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.488761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:37.489717 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:37.489727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.490453 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.490456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:37.490535 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.490552 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:50:37.490618 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:50:37.490714 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":42,"thread_start_us":114,"threads_started":1}
I20260924 04:50:37.491624 17707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638e5aba56ab49d489bd26ce6a5bc64e" candidate_uuid: "1053f166658e4ea58309d5d625a3cacb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58349eb542324ec9bdf95ad7930333d6" is_pre_election: true
I20260924 04:50:37.491624 17573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638e5aba56ab49d489bd26ce6a5bc64e" candidate_uuid: "1053f166658e4ea58309d5d625a3cacb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5" is_pre_election: true
I20260924 04:50:37.491992 17573 raft_consensus.cc:2454] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1053f166658e4ea58309d5d625a3cacb in term 0.
I20260924 04:50:37.491992 17707 raft_consensus.cc:2454] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1053f166658e4ea58309d5d625a3cacb in term 0.
I20260924 04:50:37.492479 17774 leader_election.cc:302] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053f166658e4ea58309d5d625a3cacb, 63820ccf6c8740ed8ed6b3cd48b6afc5; no voters:
I20260924 04:50:37.492738 17910 raft_consensus.cc:2781] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:50:37.492816 17910 raft_consensus.cc:482] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:50:37.492846 17910 raft_consensus.cc:3037] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:37.494215 17910 raft_consensus.cc:504] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35037 } } peers { permanent_uuid: "1053f166658e4ea58309d5d625a3cacb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38891 } } peers { permanent_uuid: "58349eb542324ec9bdf95ad7930333d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36409 } }
I20260924 04:50:37.494572 17910 leader_election.cc:288] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [CANDIDATE]: Term 1 election: Requested vote from peers 63820ccf6c8740ed8ed6b3cd48b6afc5 (127.0.98.1:35037), 58349eb542324ec9bdf95ad7930333d6 (127.0.98.2:36409)
I20260924 04:50:37.494963 17707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638e5aba56ab49d489bd26ce6a5bc64e" candidate_uuid: "1053f166658e4ea58309d5d625a3cacb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58349eb542324ec9bdf95ad7930333d6"
I20260924 04:50:37.494997 17573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638e5aba56ab49d489bd26ce6a5bc64e" candidate_uuid: "1053f166658e4ea58309d5d625a3cacb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5"
I20260924 04:50:37.495124 17573 raft_consensus.cc:3037] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:37.495126 17707 raft_consensus.cc:3037] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:37.496202 17573 raft_consensus.cc:2454] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053f166658e4ea58309d5d625a3cacb in term 1.
I20260924 04:50:37.496217 17707 raft_consensus.cc:2454] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1053f166658e4ea58309d5d625a3cacb in term 1.
I20260924 04:50:37.496531 17777 leader_election.cc:302] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1053f166658e4ea58309d5d625a3cacb, 58349eb542324ec9bdf95ad7930333d6; no voters:
I20260924 04:50:37.496719 17910 raft_consensus.cc:2781] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:37.496937 17910 raft_consensus.cc:686] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [term 1 LEADER]: Becoming Leader. State: Replica: 1053f166658e4ea58309d5d625a3cacb, State: Running, Role: LEADER
I20260924 04:50:37.497363 17910 consensus_queue.cc:237] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35037 } } peers { permanent_uuid: "1053f166658e4ea58309d5d625a3cacb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38891 } } peers { permanent_uuid: "58349eb542324ec9bdf95ad7930333d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36409 } }
I20260924 04:50:37.501191 17427 catalog_manager.cc:5495] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb reported cstate change: term changed from 0 to 1, leader changed from <none> to 1053f166658e4ea58309d5d625a3cacb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1053f166658e4ea58309d5d625a3cacb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1053f166658e4ea58309d5d625a3cacb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58349eb542324ec9bdf95ad7930333d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36409 } health_report { overall_health: UNKNOWN } } }
I20260924 04:50:37.521507 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.518111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.0.98.3:38891)
0924 04:50:37.518217 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:37.518221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:37.518234 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:50:37.518329 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:37.518331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:37.518391 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:37.518479 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:50:37.518752 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.519329 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.520188 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.520395 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.520466 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.520717 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:50:37.521147 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:50:37.521234 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:37.521306 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 04:50:37.521365 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:50:37.527722 17573 raft_consensus.cc:1261] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5 [term 1 FOLLOWER]: Refusing update from remote peer 1053f166658e4ea58309d5d625a3cacb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:50:37.527722 17707 raft_consensus.cc:1261] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6 [term 1 FOLLOWER]: Refusing update from remote peer 1053f166658e4ea58309d5d625a3cacb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:50:37.528609 17914 consensus_queue.cc:1035] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [LEADER]: Connected to new peer: Peer: permanent_uuid: "58349eb542324ec9bdf95ad7930333d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:50:37.528807 17910 consensus_queue.cc:1035] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [LEADER]: Connected to new peer: Peer: permanent_uuid: "63820ccf6c8740ed8ed6b3cd48b6afc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:50:37.538354 17918 mvcc.cc:205] Tried to move back new op lower bound from 7332763392105357312 to 7332763391992479744. Current Snapshot: MvccSnapshot[applied={T|T < 7332763392105357312}]
I20260924 04:50:37.541603 17919 mvcc.cc:205] Tried to move back new op lower bound from 7332763392105357312 to 7332763391992479744. Current Snapshot: MvccSnapshot[applied={T|T < 7332763392105357312}]
I20260924 04:50:37.547314 17427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:50:37.547487 17427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:50:37.550072 17427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6f2a39f286e43b1a9258f5003e82132: Sending DeleteTablet for 3 replicas of tablet 638e5aba56ab49d489bd26ce6a5bc64e
I20260924 04:50:37.550947 17553 tablet_service.cc:1514] Processing DeleteTablet for tablet 638e5aba56ab49d489bd26ce6a5bc64e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37494
I20260924 04:50:37.551100 17821 tablet_service.cc:1514] Processing DeleteTablet for tablet 638e5aba56ab49d489bd26ce6a5bc64e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59276
I20260924 04:50:37.551540 17926 tablet_replica.cc:330] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb: stopping tablet replica
I20260924 04:50:37.551739 17687 tablet_service.cc:1514] Processing DeleteTablet for tablet 638e5aba56ab49d489bd26ce6a5bc64e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55096
I20260924 04:50:37.551869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 17490
I20260924 04:50:37.551817 17927 tablet_replica.cc:330] T 638e5aba56ab49d489bd26ce6a5bc64e P 63820ccf6c8740ed8ed6b3cd48b6afc5: stopping tablet replica
I20260924 04:50:37.552614 17926 raft_consensus.cc:2229] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:50:37.553216 17926 raft_consensus.cc:2258] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:50:37.554629 17928 tablet_replica.cc:330] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6: stopping tablet replica
I20260924 04:50:37.555011 17928 raft_consensus.cc:2229] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:50:37.555177 17926 ts_tablet_manager.cc:1905] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:50:37.555353 17928 raft_consensus.cc:2258] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:50:37.556838 17928 ts_tablet_manager.cc:1905] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:50:37.558301 17926 ts_tablet_manager.cc:1918] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:50:37.558463 17926 log.cc:1196] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/638e5aba56ab49d489bd26ce6a5bc64e
I20260924 04:50:37.558871 17926 ts_tablet_manager.cc:1939] T 638e5aba56ab49d489bd26ce6a5bc64e P 1053f166658e4ea58309d5d625a3cacb: Deleting consensus metadata
W20260924 04:50:37.559208 17412 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35037 (error 108)
I20260924 04:50:37.559885 17928 ts_tablet_manager.cc:1918] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:50:37.559994 17928 log.cc:1196] T 638e5aba56ab49d489bd26ce6a5bc64e P 58349eb542324ec9bdf95ad7930333d6: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/638e5aba56ab49d489bd26ce6a5bc64e
I20260924 04:50:37.560119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 17624
I20260924 04:50:37.561323 17415 catalog_manager.cc:4841] TS 1053f166658e4ea58309d5d625a3cacb (127.0.98.3:38891): tablet 638e5aba56ab49d489bd26ce6a5bc64e (table test-table [id=ec51a4a79dd748df993a6cf87ee0e097]) successfully deleted
I20260924 04:50:37.563925 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.563595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35037 (local address 127.0.0.1:37502)
0924 04:50:37.563734 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:37.563813 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 04:50:37.564133 17412 catalog_manager.cc:4584] TS 63820ccf6c8740ed8ed6b3cd48b6afc5 (127.0.98.1:35037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 638e5aba56ab49d489bd26ce6a5bc64e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35037: connect: Connection refused (error 111)
I20260924 04:50:37.565692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 17758
I20260924 04:50:37.567183 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.565891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36409 (local address 127.0.0.1:55098)
0924 04:50:37.567038 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:37.567096 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1050}
I20260924 04:50:37.570768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 17396
2026-09-24T04:50:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2192 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T04:50:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:37Z Disabled control of system clock
I20260924 04:50:37.599452 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37681
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:50:37.711102 17933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:37.711393 17933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:37.711431 17933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:37.715579 17933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:37.715647 17933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:37.715662 17933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:37.715678 17933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:37.715692 17933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:37.720602 17933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:37.721671 17933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:37.723042 17933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:37.731281 17941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:37.731276 17938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:37.731279 17939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:37.731554 17933 server_base.cc:1034] running on GCE node
I20260924 04:50:37.732120 17933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:37.733354 17933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:37.734515 17933 hybrid_clock.cc:648] HybridClock initialized: now 1790225437734486 us; error 43 us; skew 500 ppm
I20260924 04:50:37.736645 17933 webserver.cc:460] Webserver started at http://127.0.98.62:37503/ using document root <none> and password file <none>
I20260924 04:50:37.737258 17933 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:37.737337 17933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:37.737528 17933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:37.739784 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "b4620c74919f4b5db9ff8b653ed1bedd"
format_stamp: "Formatted at 2026-09-24 04:50:37 on dist-test-slave-hvcl"
server_key: "fb6193df0b89e987ef7241a5bf0bbf17"
server_key_iv: "d316371e7bc6343f6dd8c393cfff20fa"
server_key_version: "encryptionkey@0"
I20260924 04:50:37.740353 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "b4620c74919f4b5db9ff8b653ed1bedd"
format_stamp: "Formatted at 2026-09-24 04:50:37 on dist-test-slave-hvcl"
server_key: "fb6193df0b89e987ef7241a5bf0bbf17"
server_key_iv: "d316371e7bc6343f6dd8c393cfff20fa"
server_key_version: "encryptionkey@0"
I20260924 04:50:37.744087 17933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:50:37.746577 17947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:37.747889 17933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:50:37.748003 17933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "b4620c74919f4b5db9ff8b653ed1bedd"
format_stamp: "Formatted at 2026-09-24 04:50:37 on dist-test-slave-hvcl"
server_key: "fb6193df0b89e987ef7241a5bf0bbf17"
server_key_iv: "d316371e7bc6343f6dd8c393cfff20fa"
server_key_version: "encryptionkey@0"
I20260924 04:50:37.748099 17933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:37.760633 17933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:37.761365 17933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:37.761546 17933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:37.768414 17933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37681
I20260924 04:50:37.768450 17999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37681 every 8 connection(s)
I20260924 04:50:37.769369 17933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:37.772526 18000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:37.775288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 17933
I20260924 04:50:37.775413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:37.775681 392 external_mini_cluster.cc:1428] Setting key d14bb9f521a3c3adc5586b8f9521953d
I20260924 04:50:37.778599 18000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd: Bootstrap starting.
I20260924 04:50:37.780826 18000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:37.781533 18000 log.cc:824] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:37.783478 18000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd: No bootstrap required, opened a new log
I20260924 04:50:37.784873 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:37.777113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.0.98.62:37681)
0924 04:50:37.777802 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:37.777819 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:37.777855 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:50:37.778493 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:37.778510 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:37.778932 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:37.779315 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:37.779738 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.780707 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.781586 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:37.782081 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:37.782225 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:37.783074 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:37.783080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:37.783092 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:37.783507 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:37.783563 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:37.783590 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:37.783679 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:37.783850 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 04:50:37.784122 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":138,"threads_started":1}
I20260924 04:50:37.787362 18000 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4620c74919f4b5db9ff8b653ed1bedd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37681 } }
I20260924 04:50:37.787592 18000 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:37.787648 18000 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4620c74919f4b5db9ff8b653ed1bedd, State: Initialized, Role: FOLLOWER
I20260924 04:50:37.788050 18000 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4620c74919f4b5db9ff8b653ed1bedd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37681 } }
I20260924 04:50:37.788177 18000 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:37.788236 18000 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:37.788349 18000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:37.789247 18000 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4620c74919f4b5db9ff8b653ed1bedd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37681 } }
I20260924 04:50:37.789601 18000 leader_election.cc:302] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4620c74919f4b5db9ff8b653ed1bedd; no voters:
I20260924 04:50:37.789896 18000 leader_election.cc:288] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:37.789991 18005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:37.790215 18005 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [term 1 LEADER]: Becoming Leader. State: Replica: b4620c74919f4b5db9ff8b653ed1bedd, State: Running, Role: LEADER
I20260924 04:50:37.790504 18005 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4620c74919f4b5db9ff8b653ed1bedd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37681 } }
I20260924 04:50:37.791079 18000 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:37.793098 18007 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4620c74919f4b5db9ff8b653ed1bedd. Latest consensus state: current_term: 1 leader_uuid: "b4620c74919f4b5db9ff8b653ed1bedd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4620c74919f4b5db9ff8b653ed1bedd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37681 } } }
I20260924 04:50:37.793236 18007 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:37.793208 18006 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4620c74919f4b5db9ff8b653ed1bedd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4620c74919f4b5db9ff8b653ed1bedd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37681 } } }
I20260924 04:50:37.793320 18006 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd [sys.catalog]: This master's current role is: LEADER
W20260924 04:50:37.796065 18021 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:50:37.796139 18021 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:50:37.796211 18020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:37.797034 18020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:37.802950 18020 catalog_manager.cc:1347] Generated new cluster ID: d66d1d0eabe848dfbd058982aa9d675c
I20260924 04:50:37.803045 18020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:37.839148 18020 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:37.840517 18020 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:37.848819 18020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd: Generated new TSK 0
I20260924 04:50:37.849522 18020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:50:37.861512 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37681
--builtin_ntp_servers=127.0.98.20:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:50:37.972609 18024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:37.972898 18024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:37.972937 18024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:37.977151 18024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:37.977226 18024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:37.977285 18024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:50:37.982275 18024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:37.983515 18024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:37.984968 18024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:50:37.994961 18024 server_base.cc:1034] running on GCE node
W20260924 04:50:37.994944 18032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:37.994995 18030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:37.994994 18029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:37.995618 18024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:37.996299 18024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:37.997484 18024 hybrid_clock.cc:648] HybridClock initialized: now 1790225437997463 us; error 36 us; skew 500 ppm
I20260924 04:50:37.999511 18024 webserver.cc:460] Webserver started at http://127.0.98.1:33693/ using document root <none> and password file <none>
I20260924 04:50:38.000102 18024 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:38.000172 18024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:38.000392 18024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:38.002702 18024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "95aad1d411324526a4915334ad951df5"
format_stamp: "Formatted at 2026-09-24 04:50:37 on dist-test-slave-hvcl"
server_key: "ea34ef5b4468b85876a104657f955dd6"
server_key_iv: "c53627bad88e27efbe0327e3b0970fc2"
server_key_version: "encryptionkey@0"
I20260924 04:50:38.003409 18024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "95aad1d411324526a4915334ad951df5"
format_stamp: "Formatted at 2026-09-24 04:50:37 on dist-test-slave-hvcl"
server_key: "ea34ef5b4468b85876a104657f955dd6"
server_key_iv: "c53627bad88e27efbe0327e3b0970fc2"
server_key_version: "encryptionkey@0"
I20260924 04:50:38.007364 18024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:50:38.010056 18038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:38.011524 18024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:50:38.011657 18024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "95aad1d411324526a4915334ad951df5"
format_stamp: "Formatted at 2026-09-24 04:50:37 on dist-test-slave-hvcl"
server_key: "ea34ef5b4468b85876a104657f955dd6"
server_key_iv: "c53627bad88e27efbe0327e3b0970fc2"
server_key_version: "encryptionkey@0"
I20260924 04:50:38.011776 18024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:38.034512 18024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:38.035265 18024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:38.035444 18024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:38.036051 18024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:38.037032 18024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:38.037078 18024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:38.037132 18024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:38.037158 18024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:38.045637 18024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38887
I20260924 04:50:38.045660 18151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38887 every 8 connection(s)
I20260924 04:50:38.046535 18024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:50:38.047781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 18024
I20260924 04:50:38.047907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:50:38.048233 392 external_mini_cluster.cc:1428] Setting key c01ec5716e4292725c8b2e4f55bf77fc
I20260924 04:50:38.050320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37681
--builtin_ntp_servers=127.0.98.20:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:50:38.057588 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.048403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48171 (local address 127.0.98.62:37681)
0924 04:50:38.048530 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:38.048533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:38.049795 (+ 1262us) server_negotiation.cc:407] Connection header received
0924 04:50:38.051599 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:38.051602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:38.051674 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:38.051770 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:38.053148 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.053693 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.055433 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.055653 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.055818 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.057097 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:38.057100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:38.057102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:38.057115 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:38.057135 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:38.057143 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:38.057203 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:38.057407 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 04:50:38.057465 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:50:38.058866 18154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.048875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37681 (local address 127.0.98.1:48171)
0924 04:50:38.049643 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:38.049696 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:38.051342 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:38.052165 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:38.052175 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:38.052295 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:38.053005 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.053023 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.053820 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.053824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.055278 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.055295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.055878 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.055882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.056097 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.056761 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:50:38.056763 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:38.056828 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:50:38.056922 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:38.057234 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:38.057243 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:38.057250 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:50:38.057396 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 04:50:38.057748 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":151,"threads_started":1}
I20260924 04:50:38.060197 18152 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37681
I20260924 04:50:38.060423 18152 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:38.061000 18152 heartbeater.cc:511] Master 127.0.98.62:37681 requested a full tablet report, sending...
I20260924 04:50:38.062590 17964 ts_manager.cc:194] Registered new tserver with Master: 95aad1d411324526a4915334ad951df5 (127.0.98.1:38887)
I20260924 04:50:38.064198 17964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48171
W20260924 04:50:38.164685 18155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:38.164968 18155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:38.165006 18155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:38.170113 18155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:38.170183 18155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:38.170241 18155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:50:38.175155 18155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:38.176239 18155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:38.177625 18155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:38.187202 18161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:38.187280 18155 server_base.cc:1034] running on GCE node
W20260924 04:50:38.187188 18160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:38.187188 18163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:38.187799 18155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:38.188392 18155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:38.189551 18155 hybrid_clock.cc:648] HybridClock initialized: now 1790225438189526 us; error 39 us; skew 500 ppm
I20260924 04:50:38.191514 18155 webserver.cc:460] Webserver started at http://127.0.98.2:45889/ using document root <none> and password file <none>
I20260924 04:50:38.192137 18155 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:38.192201 18155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:38.192373 18155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:38.194352 18155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "6bd3a288a4d54e6fb58cf6716b0137b1"
format_stamp: "Formatted at 2026-09-24 04:50:38 on dist-test-slave-hvcl"
server_key: "53a2af360d9fd46b66238d9a7556fa8f"
server_key_iv: "96dd455dfbd0ffd613a27a0d12651839"
server_key_version: "encryptionkey@0"
I20260924 04:50:38.194844 18155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6bd3a288a4d54e6fb58cf6716b0137b1"
format_stamp: "Formatted at 2026-09-24 04:50:38 on dist-test-slave-hvcl"
server_key: "53a2af360d9fd46b66238d9a7556fa8f"
server_key_iv: "96dd455dfbd0ffd613a27a0d12651839"
server_key_version: "encryptionkey@0"
I20260924 04:50:38.198383 18155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:50:38.200703 18169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:38.201936 18155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:50:38.202036 18155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "6bd3a288a4d54e6fb58cf6716b0137b1"
format_stamp: "Formatted at 2026-09-24 04:50:38 on dist-test-slave-hvcl"
server_key: "53a2af360d9fd46b66238d9a7556fa8f"
server_key_iv: "96dd455dfbd0ffd613a27a0d12651839"
server_key_version: "encryptionkey@0"
I20260924 04:50:38.202127 18155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:38.218322 18155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:38.219100 18155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:38.219267 18155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:38.219834 18155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:38.220739 18155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:38.220785 18155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:38.220836 18155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:38.220868 18155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:38.228745 18155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44171
I20260924 04:50:38.228842 18282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44171 every 8 connection(s)
I20260924 04:50:38.229672 18155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:50:38.237483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 18155
I20260924 04:50:38.237625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:50:38.237911 392 external_mini_cluster.cc:1428] Setting key 7988851c27b5fe414c09a7b05f7cd0a5
I20260924 04:50:38.239148 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.231561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44579 (local address 127.0.98.62:37681)
0924 04:50:38.231712 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:38.231717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:38.232834 (+ 1117us) server_negotiation.cc:407] Connection header received
0924 04:50:38.234212 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:38.234215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:38.234266 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:38.234341 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:38.235407 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.235959 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.237144 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.237368 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.237459 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.238646 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:38.238650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:38.238653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:38.238668 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:38.238692 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:38.238701 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:38.238781 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:38.238921 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 04:50:38.239010 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:50:38.240023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37681
--builtin_ntp_servers=127.0.98.20:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:50:38.240083 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.232014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37681 (local address 127.0.98.2:44579)
0924 04:50:38.232680 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:38.232723 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:38.234004 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:38.234571 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:38.234584 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:38.234678 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:38.235230 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.235246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.236096 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.236099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.237023 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.237034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.237529 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.237534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.237712 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.238315 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:50:38.238317 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:38.238367 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:50:38.238456 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:38.238781 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:38.238788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:38.238792 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:50:38.238897 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 04:50:38.239202 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":113,"threads_started":1}
I20260924 04:50:38.241168 18283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37681
I20260924 04:50:38.241436 18283 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:38.241940 18283 heartbeater.cc:511] Master 127.0.98.62:37681 requested a full tablet report, sending...
I20260924 04:50:38.242995 17964 ts_manager.cc:194] Registered new tserver with Master: 6bd3a288a4d54e6fb58cf6716b0137b1 (127.0.98.2:44171)
I20260924 04:50:38.243551 17964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44579
W20260924 04:50:38.351248 18286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:38.351531 18286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:38.351588 18286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:38.355935 18286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:38.356005 18286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:38.356086 18286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:50:38.361167 18286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:38.362367 18286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:38.363888 18286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:38.373797 18294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:38.373857 18291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:38.374008 18286 server_base.cc:1034] running on GCE node
W20260924 04:50:38.373801 18292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:38.374501 18286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:38.375216 18286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:38.376411 18286 hybrid_clock.cc:648] HybridClock initialized: now 1790225438376384 us; error 38 us; skew 500 ppm
I20260924 04:50:38.378415 18286 webserver.cc:460] Webserver started at http://127.0.98.3:41585/ using document root <none> and password file <none>
I20260924 04:50:38.379030 18286 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:38.379109 18286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:38.379309 18286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:38.381425 18286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "540fae019fd243759dd99f255301df69"
format_stamp: "Formatted at 2026-09-24 04:50:38 on dist-test-slave-hvcl"
server_key: "51eacbe779c2269bf2bbd4dc131a1a8f"
server_key_iv: "b83ad7259312c843f6159411f1cf570e"
server_key_version: "encryptionkey@0"
I20260924 04:50:38.381981 18286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "540fae019fd243759dd99f255301df69"
format_stamp: "Formatted at 2026-09-24 04:50:38 on dist-test-slave-hvcl"
server_key: "51eacbe779c2269bf2bbd4dc131a1a8f"
server_key_iv: "b83ad7259312c843f6159411f1cf570e"
server_key_version: "encryptionkey@0"
I20260924 04:50:38.385747 18286 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:50:38.388295 18300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:38.389556 18286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:50:38.389669 18286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "540fae019fd243759dd99f255301df69"
format_stamp: "Formatted at 2026-09-24 04:50:38 on dist-test-slave-hvcl"
server_key: "51eacbe779c2269bf2bbd4dc131a1a8f"
server_key_iv: "b83ad7259312c843f6159411f1cf570e"
server_key_version: "encryptionkey@0"
I20260924 04:50:38.389825 18286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:38.420974 18286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:38.421748 18286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:38.421962 18286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:38.422601 18286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:38.423655 18286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:38.423708 18286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:38.423774 18286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:38.423799 18286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:38.432514 18286 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42987
I20260924 04:50:38.432560 18413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42987 every 8 connection(s)
I20260924 04:50:38.433455 18286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:50:38.436476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 18286
I20260924 04:50:38.436600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:50:38.436904 392 external_mini_cluster.cc:1428] Setting key 7bc0e1cd53e80cb1d891fef6393030a5
I20260924 04:50:38.443125 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.435479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44573 (local address 127.0.98.62:37681)
0924 04:50:38.435645 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:38.435650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:38.436953 (+ 1303us) server_negotiation.cc:407] Connection header received
0924 04:50:38.438349 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:38.438353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:38.438406 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:38.438482 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:38.439492 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.440038 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.441188 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.441418 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.441503 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.442607 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:38.442610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:38.442612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:38.442625 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:38.442643 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:38.442651 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:38.442723 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:38.442874 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 04:50:38.442947 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:50:38.444048 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.435946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37681 (local address 127.0.98.3:44573)
0924 04:50:38.436784 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:38.436832 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:38.438114 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:38.438682 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:38.438689 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:38.438780 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:38.439355 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.439369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.440155 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.440158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.441064 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.441073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.441560 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.441563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.441728 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.442338 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:50:38.442340 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:38.442385 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:50:38.442454 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:38.442735 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:38.442745 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:38.442751 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:50:38.442865 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 04:50:38.443234 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":153,"threads_started":1}
I20260924 04:50:38.445291 18414 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37681
I20260924 04:50:38.445521 18414 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:38.446036 18414 heartbeater.cc:511] Master 127.0.98.62:37681 requested a full tablet report, sending...
I20260924 04:50:38.447084 17964 ts_manager.cc:194] Registered new tserver with Master: 540fae019fd243759dd99f255301df69 (127.0.98.3:42987)
I20260924 04:50:38.447669 17964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44573
I20260924 04:50:38.450632 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:50:38.455866 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.452599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.0.98.62:37681)
0924 04:50:38.452723 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:38.452728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:38.452871 (+ 143us) server_negotiation.cc:407] Connection header received
0924 04:50:38.452936 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:38.452938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:38.452981 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:38.453050 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:38.453255 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.453795 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.454577 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.454775 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.454839 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.455545 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:38.455548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:38.455550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:38.455561 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:38.455579 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:38.455587 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:38.455640 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:38.455702 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 04:50:38.455751 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:50:38.458528 17964 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:50:38.461566 17964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:50:38.476042 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.470425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.0.98.3:42987)
0924 04:50:38.470901 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:38.470913 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:38.470937 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:50:38.471034 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:38.471040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:38.471669 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:38.471822 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:38.472282 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.473053 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.474493 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.475319 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.475427 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.475628 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:38.475761 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 04:50:38.475867 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":134,"threads_started":1}
I20260924 04:50:38.476038 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.470244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44171 (local address 127.0.0.1:60152)
0924 04:50:38.471139 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:38.471151 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:38.471242 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:38.471924 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:38.471930 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:38.471953 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:38.472148 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.472156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.473624 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.473627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.474644 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.474654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.475663 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.475667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.475756 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.475780 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:50:38.475826 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 04:50:38.475889 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":50,"spinlock_wait_cycles":1920,"thread_start_us":128,"threads_started":1}
I20260924 04:50:38.476042 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.470830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38887 (local address 127.0.0.1:33464)
0924 04:50:38.471393 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:38.471405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:38.471468 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:38.472132 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:38.472134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:38.472149 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:38.472232 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.472238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.473590 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.473594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.474765 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.474775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.475619 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.475622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.475710 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.475745 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:50:38.475785 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 04:50:38.475846 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":65,"thread_start_us":92,"threads_started":1}
I20260924 04:50:38.476042 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.470117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42987 (local address 127.0.0.1:48074)
0924 04:50:38.470649 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:38.470706 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:38.470837 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:38.471975 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:38.471977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:38.472007 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:38.472156 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.472163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.473154 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.473157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.474374 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.474384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.475441 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.475444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.475559 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.475605 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:50:38.475713 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 04:50:38.475843 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":36,"spinlock_wait_cycles":1664,"thread_start_us":83,"threads_started":1}
I20260924 04:50:38.476878 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.470804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.0.98.2:44171)
0924 04:50:38.471344 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:38.471355 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:38.471376 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:50:38.471444 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:38.471450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:38.471657 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:38.471910 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:38.472273 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.473476 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.474776 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.475538 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.476420 (+ 882us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.476577 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:38.476627 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 04:50:38.476721 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":89,"threads_started":1}
I20260924 04:50:38.477306 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.471233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.0.98.1:38887)
0924 04:50:38.471709 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:38.471720 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:38.471742 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:50:38.471812 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:38.471816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:38.471997 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:38.472150 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:38.472335 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.473477 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.474901 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.475494 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.476753 (+ 1259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.477046 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:38.477099 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 04:50:38.477184 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":137,"threads_started":1}
I20260924 04:50:38.479755 18086 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3c224dd337433184a5e08a0c553f31 (DEFAULT_TABLE table=test-table [id=d219fdec64ad49399bd798c352ba0f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:38.479780 18348 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3c224dd337433184a5e08a0c553f31 (DEFAULT_TABLE table=test-table [id=d219fdec64ad49399bd798c352ba0f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:38.479776 18217 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3c224dd337433184a5e08a0c553f31 (DEFAULT_TABLE table=test-table [id=d219fdec64ad49399bd798c352ba0f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:38.480742 18086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3c224dd337433184a5e08a0c553f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:38.480754 18348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3c224dd337433184a5e08a0c553f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:38.480754 18217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3c224dd337433184a5e08a0c553f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:38.487336 18431 tablet_bootstrap.cc:492] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5: Bootstrap starting.
I20260924 04:50:38.488593 18433 tablet_bootstrap.cc:492] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1: Bootstrap starting.
I20260924 04:50:38.489437 18431 tablet_bootstrap.cc:654] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:38.490140 18431 log.cc:824] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:38.490363 18433 tablet_bootstrap.cc:654] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:38.490841 18432 tablet_bootstrap.cc:492] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69: Bootstrap starting.
I20260924 04:50:38.491196 18433 log.cc:824] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:38.492933 18431 tablet_bootstrap.cc:492] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5: No bootstrap required, opened a new log
I20260924 04:50:38.493119 18431 ts_tablet_manager.cc:1397] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 04:50:38.493278 18433 tablet_bootstrap.cc:492] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1: No bootstrap required, opened a new log
I20260924 04:50:38.493505 18433 ts_tablet_manager.cc:1397] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260924 04:50:38.493542 18432 tablet_bootstrap.cc:654] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:38.494357 18432 log.cc:824] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:38.496527 18432 tablet_bootstrap.cc:492] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69: No bootstrap required, opened a new log
I20260924 04:50:38.496735 18432 ts_tablet_manager.cc:1397] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:50:38.496848 18431 raft_consensus.cc:348] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540fae019fd243759dd99f255301df69" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42987 } } peers { permanent_uuid: "6bd3a288a4d54e6fb58cf6716b0137b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44171 } } peers { permanent_uuid: "95aad1d411324526a4915334ad951df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38887 } }
I20260924 04:50:38.497092 18431 raft_consensus.cc:374] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:38.497145 18431 raft_consensus.cc:729] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95aad1d411324526a4915334ad951df5, State: Initialized, Role: FOLLOWER
I20260924 04:50:38.497582 18431 consensus_queue.cc:260] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540fae019fd243759dd99f255301df69" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42987 } } peers { permanent_uuid: "6bd3a288a4d54e6fb58cf6716b0137b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44171 } } peers { permanent_uuid: "95aad1d411324526a4915334ad951df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38887 } }
I20260924 04:50:38.497538 18433 raft_consensus.cc:348] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540fae019fd243759dd99f255301df69" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42987 } } peers { permanent_uuid: "6bd3a288a4d54e6fb58cf6716b0137b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44171 } } peers { permanent_uuid: "95aad1d411324526a4915334ad951df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38887 } }
I20260924 04:50:38.497802 18433 raft_consensus.cc:374] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:38.497859 18433 raft_consensus.cc:729] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd3a288a4d54e6fb58cf6716b0137b1, State: Initialized, Role: FOLLOWER
I20260924 04:50:38.498270 18433 consensus_queue.cc:260] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540fae019fd243759dd99f255301df69" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42987 } } peers { permanent_uuid: "6bd3a288a4d54e6fb58cf6716b0137b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44171 } } peers { permanent_uuid: "95aad1d411324526a4915334ad951df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38887 } }
I20260924 04:50:38.498517 18431 ts_tablet_manager.cc:1428] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:50:38.498905 18152 heartbeater.cc:503] Master 127.0.98.62:37681 was elected leader, sending a full tablet report...
I20260924 04:50:38.499064 18283 heartbeater.cc:503] Master 127.0.98.62:37681 was elected leader, sending a full tablet report...
I20260924 04:50:38.500273 18433 ts_tablet_manager.cc:1428] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260924 04:50:38.501428 18432 raft_consensus.cc:348] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540fae019fd243759dd99f255301df69" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42987 } } peers { permanent_uuid: "6bd3a288a4d54e6fb58cf6716b0137b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44171 } } peers { permanent_uuid: "95aad1d411324526a4915334ad951df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38887 } }
I20260924 04:50:38.501644 18432 raft_consensus.cc:374] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:38.501693 18432 raft_consensus.cc:729] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540fae019fd243759dd99f255301df69, State: Initialized, Role: FOLLOWER
I20260924 04:50:38.502086 18432 consensus_queue.cc:260] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540fae019fd243759dd99f255301df69" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42987 } } peers { permanent_uuid: "6bd3a288a4d54e6fb58cf6716b0137b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44171 } } peers { permanent_uuid: "95aad1d411324526a4915334ad951df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38887 } }
I20260924 04:50:38.502729 18414 heartbeater.cc:503] Master 127.0.98.62:37681 was elected leader, sending a full tablet report...
I20260924 04:50:38.503108 18432 ts_tablet_manager.cc:1428] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260924 04:50:38.547731 18153 tablet.cc:2369] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:50:38.573117 18437 raft_consensus.cc:482] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:50:38.573323 18437 raft_consensus.cc:504] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540fae019fd243759dd99f255301df69" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42987 } } peers { permanent_uuid: "6bd3a288a4d54e6fb58cf6716b0137b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44171 } } peers { permanent_uuid: "95aad1d411324526a4915334ad951df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38887 } }
I20260924 04:50:38.574517 18437 leader_election.cc:288] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 540fae019fd243759dd99f255301df69 (127.0.98.3:42987), 6bd3a288a4d54e6fb58cf6716b0137b1 (127.0.98.2:44171)
I20260924 04:50:38.578143 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.574849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45199 (local address 127.0.98.3:42987)
0924 04:50:38.575015 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:38.575020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:38.575144 (+ 124us) server_negotiation.cc:407] Connection header received
0924 04:50:38.575280 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:38.575283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:38.575322 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:38.575383 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:38.575636 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.576128 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.577190 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.577577 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.577665 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.577724 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:38.577955 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:50:38.578016 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:50:38.578145 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.574929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35111 (local address 127.0.98.2:44171)
0924 04:50:38.575081 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:38.575086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:38.575237 (+ 151us) server_negotiation.cc:407] Connection header received
0924 04:50:38.575377 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:38.575379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:38.575419 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:38.575479 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:38.575706 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.576188 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.577197 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.577579 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.577666 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.577734 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:38.577960 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 04:50:38.578026 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 04:50:38.578176 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.574737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42987 (local address 127.0.98.1:45199)
0924 04:50:38.575060 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:38.575073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:38.575180 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:38.575408 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:38.575411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:38.575425 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:38.575544 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.575549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.576230 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.576233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.577077 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.577086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.577685 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.577689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.577816 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.577869 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 04:50:38.577930 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 04:50:38.578001 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":51,"thread_start_us":69,"threads_started":1}
I20260924 04:50:38.578177 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.574793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44171 (local address 127.0.98.1:35111)
0924 04:50:38.575155 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:38.575165 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:38.575285 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:38.575494 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:38.575497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:38.575512 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:38.575614 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.575619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.576284 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.576287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.577091 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:38.577099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.577680 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.577684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:38.577834 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.577874 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 04:50:38.577936 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:50:38.578002 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":92,"thread_start_us":73,"threads_started":1}
I20260924 04:50:38.578727 18368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3c224dd337433184a5e08a0c553f31" candidate_uuid: "95aad1d411324526a4915334ad951df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540fae019fd243759dd99f255301df69" is_pre_election: true
I20260924 04:50:38.578888 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3c224dd337433184a5e08a0c553f31" candidate_uuid: "95aad1d411324526a4915334ad951df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd3a288a4d54e6fb58cf6716b0137b1" is_pre_election: true
I20260924 04:50:38.579046 18368 raft_consensus.cc:2454] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95aad1d411324526a4915334ad951df5 in term 0.
I20260924 04:50:38.579177 18237 raft_consensus.cc:2454] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95aad1d411324526a4915334ad951df5 in term 0.
I20260924 04:50:38.579496 18042 leader_election.cc:302] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 540fae019fd243759dd99f255301df69, 95aad1d411324526a4915334ad951df5; no voters:
I20260924 04:50:38.579739 18437 raft_consensus.cc:2781] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:50:38.579818 18437 raft_consensus.cc:482] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:50:38.579871 18437 raft_consensus.cc:3037] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:38.580824 18437 raft_consensus.cc:504] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540fae019fd243759dd99f255301df69" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42987 } } peers { permanent_uuid: "6bd3a288a4d54e6fb58cf6716b0137b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44171 } } peers { permanent_uuid: "95aad1d411324526a4915334ad951df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38887 } }
I20260924 04:50:38.581146 18437 leader_election.cc:288] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [CANDIDATE]: Term 1 election: Requested vote from peers 540fae019fd243759dd99f255301df69 (127.0.98.3:42987), 6bd3a288a4d54e6fb58cf6716b0137b1 (127.0.98.2:44171)
I20260924 04:50:38.581447 18368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3c224dd337433184a5e08a0c553f31" candidate_uuid: "95aad1d411324526a4915334ad951df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540fae019fd243759dd99f255301df69"
I20260924 04:50:38.581468 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3c224dd337433184a5e08a0c553f31" candidate_uuid: "95aad1d411324526a4915334ad951df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd3a288a4d54e6fb58cf6716b0137b1"
I20260924 04:50:38.581552 18368 raft_consensus.cc:3037] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:38.581575 18237 raft_consensus.cc:3037] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:38.582525 18368 raft_consensus.cc:2454] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95aad1d411324526a4915334ad951df5 in term 1.
I20260924 04:50:38.582578 18237 raft_consensus.cc:2454] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95aad1d411324526a4915334ad951df5 in term 1.
I20260924 04:50:38.582793 18042 leader_election.cc:302] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 540fae019fd243759dd99f255301df69, 95aad1d411324526a4915334ad951df5; no voters:
I20260924 04:50:38.582962 18437 raft_consensus.cc:2781] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:38.583204 18437 raft_consensus.cc:686] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [term 1 LEADER]: Becoming Leader. State: Replica: 95aad1d411324526a4915334ad951df5, State: Running, Role: LEADER
I20260924 04:50:38.583537 18437 consensus_queue.cc:237] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540fae019fd243759dd99f255301df69" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42987 } } peers { permanent_uuid: "6bd3a288a4d54e6fb58cf6716b0137b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44171 } } peers { permanent_uuid: "95aad1d411324526a4915334ad951df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38887 } }
I20260924 04:50:38.586601 17963 catalog_manager.cc:5495] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95aad1d411324526a4915334ad951df5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "95aad1d411324526a4915334ad951df5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540fae019fd243759dd99f255301df69" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bd3a288a4d54e6fb58cf6716b0137b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95aad1d411324526a4915334ad951df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38887 } health_report { overall_health: HEALTHY } } }
I20260924 04:50:38.598551 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.595603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.0.98.1:38887)
0924 04:50:38.595701 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:38.595704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:38.595717 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:50:38.595818 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:38.595820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:38.595873 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:38.595953 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:50:38.596174 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.596657 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.597429 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:38.597626 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:38.597691 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:38.597893 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:50:38.598294 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:50:38.598375 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:38.598401 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 04:50:38.598443 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 04:50:38.604033 18368 raft_consensus.cc:1261] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69 [term 1 FOLLOWER]: Refusing update from remote peer 95aad1d411324526a4915334ad951df5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:50:38.604033 18237 raft_consensus.cc:1261] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1 [term 1 FOLLOWER]: Refusing update from remote peer 95aad1d411324526a4915334ad951df5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:50:38.604677 18437 consensus_queue.cc:1035] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd3a288a4d54e6fb58cf6716b0137b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:50:38.604871 18442 consensus_queue.cc:1035] T 1a3c224dd337433184a5e08a0c553f31 P 95aad1d411324526a4915334ad951df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "540fae019fd243759dd99f255301df69" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:50:38.611886 18445 mvcc.cc:205] Tried to move back new op lower bound from 7332763396515405824 to 7332763396440608768. Current Snapshot: MvccSnapshot[applied={T|T < 7332763396515405824}]
I20260924 04:50:38.612792 18447 mvcc.cc:205] Tried to move back new op lower bound from 7332763396515405824 to 7332763396440608768. Current Snapshot: MvccSnapshot[applied={T|T < 7332763396515405824}]
I20260924 04:50:38.614099 18446 mvcc.cc:205] Tried to move back new op lower bound from 7332763396515405824 to 7332763396440608768. Current Snapshot: MvccSnapshot[applied={T|T < 7332763396515405824}]
I20260924 04:50:38.622282 17962 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:50:38.622457 17962 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:50:38.624986 17962 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4620c74919f4b5db9ff8b653ed1bedd: Sending DeleteTablet for 3 replicas of tablet 1a3c224dd337433184a5e08a0c553f31
I20260924 04:50:38.625878 18217 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a3c224dd337433184a5e08a0c553f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:38 UTC) from {username='slave'} at 127.0.0.1:60152
I20260924 04:50:38.625895 18348 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a3c224dd337433184a5e08a0c553f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:38 UTC) from {username='slave'} at 127.0.0.1:48074
I20260924 04:50:38.626489 18454 tablet_replica.cc:330] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69: stopping tablet replica
I20260924 04:50:38.626600 18086 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a3c224dd337433184a5e08a0c553f31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:38 UTC) from {username='slave'} at 127.0.0.1:33464
I20260924 04:50:38.626888 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 18024
I20260924 04:50:38.627775 18454 raft_consensus.cc:2229] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:50:38.628194 18454 raft_consensus.cc:2258] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:50:38.628603 18455 tablet_replica.cc:330] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1: stopping tablet replica
I20260924 04:50:38.628934 18455 raft_consensus.cc:2229] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:50:38.629205 18455 raft_consensus.cc:2258] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:50:38.629750 18454 ts_tablet_manager.cc:1905] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:50:38.630422 18455 ts_tablet_manager.cc:1905] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:50:38.632444 18454 ts_tablet_manager.cc:1918] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:50:38.632463 18455 ts_tablet_manager.cc:1918] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:50:38.632546 18455 log.cc:1196] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/1a3c224dd337433184a5e08a0c553f31
I20260924 04:50:38.632547 18454 log.cc:1196] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/1a3c224dd337433184a5e08a0c553f31
I20260924 04:50:38.632824 18455 ts_tablet_manager.cc:1939] T 1a3c224dd337433184a5e08a0c553f31 P 6bd3a288a4d54e6fb58cf6716b0137b1: Deleting consensus metadata
I20260924 04:50:38.632831 18454 ts_tablet_manager.cc:1939] T 1a3c224dd337433184a5e08a0c553f31 P 540fae019fd243759dd99f255301df69: Deleting consensus metadata
I20260924 04:50:38.633591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 18155
W20260924 04:50:38.633996 17951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38887 (error 108)
I20260924 04:50:38.634459 17951 catalog_manager.cc:4841] TS 540fae019fd243759dd99f255301df69 (127.0.98.3:42987): tablet 1a3c224dd337433184a5e08a0c553f31 (table test-table [id=d219fdec64ad49399bd798c352ba0f9c]) successfully deleted
I20260924 04:50:38.635015 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.634705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38887 (local address 127.0.0.1:33494)
0924 04:50:38.634835 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:38.634903 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 04:50:38.635140 17951 catalog_manager.cc:4584] TS 95aad1d411324526a4915334ad951df5 (127.0.98.1:38887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a3c224dd337433184a5e08a0c553f31: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38887: connect: Connection refused (error 111)
I20260924 04:50:38.639140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 18286
I20260924 04:50:38.639592 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:38.639311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44171 (local address 127.0.0.1:60154)
0924 04:50:38.639469 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:38.639511 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 04:50:38.643965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 17933
2026-09-24T04:50:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1072 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:50:38 dist-test-slave-hvcl krb5kdc[18460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:50:38 dist-test-slave-hvcl krb5kdc[18460](info): set up 2 sockets
Sep 24 04:50:38 dist-test-slave-hvcl krb5kdc[18460](info): commencing operation
krb5kdc: starting...
W20260924 04:50:39.606593 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:50:39 dist-test-slave-hvcl krb5kdc[18460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:50:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:50:39.727236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34545
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:39.836928 18473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:39.837224 18473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:39.837285 18473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:39.841336 18473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:39.841404 18473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:39.841431 18473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:39.841452 18473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:39.841473 18473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:39.846881 18473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:39.847970 18473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:39.848980 18473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:39.857275 18481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:39.857290 18478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:39.857368 18473 server_base.cc:1034] running on GCE node
W20260924 04:50:39.857291 18479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:39.858130 18473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:39.859324 18473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:39.860481 18473 hybrid_clock.cc:648] HybridClock initialized: now 1790225439860455 us; error 42 us; skew 500 ppm
Sep 24 04:50:39 dist-test-slave-hvcl krb5kdc[18460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:39.863121 18473 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:50:39.864038 18473 webserver.cc:460] Webserver started at http://127.0.98.62:44163/ using document root <none> and password file <none>
I20260924 04:50:39.864594 18473 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:39.864661 18473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:39.864858 18473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:39.866837 18473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "5412813351584d05be3ca598febd5c47"
format_stamp: "Formatted at 2026-09-24 04:50:39 on dist-test-slave-hvcl"
server_key: "e0e13ccf9a61e45144a8bc2145e2b6a9"
server_key_iv: "30ac738169ed4df8c5f7d0f0b6019c26"
server_key_version: "encryptionkey@0"
I20260924 04:50:39.867415 18473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "5412813351584d05be3ca598febd5c47"
format_stamp: "Formatted at 2026-09-24 04:50:39 on dist-test-slave-hvcl"
server_key: "e0e13ccf9a61e45144a8bc2145e2b6a9"
server_key_iv: "30ac738169ed4df8c5f7d0f0b6019c26"
server_key_version: "encryptionkey@0"
I20260924 04:50:39.870930 18473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:50:39.873555 18488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:39.874735 18473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:50:39.874845 18473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "5412813351584d05be3ca598febd5c47"
format_stamp: "Formatted at 2026-09-24 04:50:39 on dist-test-slave-hvcl"
server_key: "e0e13ccf9a61e45144a8bc2145e2b6a9"
server_key_iv: "30ac738169ed4df8c5f7d0f0b6019c26"
server_key_version: "encryptionkey@0"
I20260924 04:50:39.874941 18473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:39.896251 18473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:39.899231 18473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:39.899417 18473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:39.906209 18473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34545
I20260924 04:50:39.906242 18540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34545 every 8 connection(s)
I20260924 04:50:39.907281 18473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:39.910080 18541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:39.913555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 18473
I20260924 04:50:39.913695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:39.913992 392 external_mini_cluster.cc:1428] Setting key cacb16e5b04bce7b6e82960b6fc89c83
I20260924 04:50:39.916235 18541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47: Bootstrap starting.
I20260924 04:50:39.918416 18541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:39.919152 18541 log.cc:824] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:39.921264 18541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47: No bootstrap required, opened a new log
Sep 24 04:50:39 dist-test-slave-hvcl krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:50:39.924682 18541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5412813351584d05be3ca598febd5c47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34545 } }
I20260924 04:50:39.924925 18541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:39.924983 18541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5412813351584d05be3ca598febd5c47, State: Initialized, Role: FOLLOWER
I20260924 04:50:39.925418 18541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5412813351584d05be3ca598febd5c47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34545 } }
I20260924 04:50:39.925546 18541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:39.925609 18541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:39.925706 18541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:39.926800 18541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5412813351584d05be3ca598febd5c47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34545 } }
I20260924 04:50:39.927166 18541 leader_election.cc:302] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5412813351584d05be3ca598febd5c47; no voters:
I20260924 04:50:39.927474 18541 leader_election.cc:288] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:39.927606 18546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:39.927837 18546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [term 1 LEADER]: Becoming Leader. State: Replica: 5412813351584d05be3ca598febd5c47, State: Running, Role: LEADER
I20260924 04:50:39.928154 18546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5412813351584d05be3ca598febd5c47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34545 } }
I20260924 04:50:39.928830 18541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:39.929325 18548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5412813351584d05be3ca598febd5c47. Latest consensus state: current_term: 1 leader_uuid: "5412813351584d05be3ca598febd5c47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5412813351584d05be3ca598febd5c47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34545 } } }
I20260924 04:50:39.929450 18548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:39.929610 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:39.915512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34200 (local address 127.0.98.62:34545)
0924 04:50:39.915959 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:39.915975 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:39.916009 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:50:39.916671 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:39.916691 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:39.917037 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:39.917339 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:39.918118 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:39.919177 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:39.920260 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:39.920728 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:39.920843 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:39.923590 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:39.923596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:39.923610 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:39.923638 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:39.925937 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:39.926426 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:39.926431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:39.926435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:39.926559 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:39.926860 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:39.926863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:39.926864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:39.927374 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:39.927571 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:39.928312 (+ 741us) server_negotiation.cc:299] Negotiation successful
0924 04:50:39.928618 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":136,"threads_started":1}
I20260924 04:50:39.929760 18547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5412813351584d05be3ca598febd5c47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5412813351584d05be3ca598febd5c47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34545 } } }
I20260924 04:50:39.929840 18547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47 [sys.catalog]: This master's current role is: LEADER
W20260924 04:50:39.933408 18561 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:50:39.933473 18561 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:50:39.933554 18551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:39.934199 18551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:39.939810 18551 catalog_manager.cc:1347] Generated new cluster ID: b8190ea33e2740dfbc025db127bba71f
I20260924 04:50:39.939882 18551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:39.956291 18551 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:39.957613 18551 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:39.967330 18551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47: Generated new TSK 0
I20260924 04:50:39.968070 18551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:50:40.002630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34545
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:40.147907 18567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:40.148185 18567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:40.148229 18567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:40.152369 18567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:40.152438 18567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:40.152509 18567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:50:40.157917 18567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:40.159021 18567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:40.160076 18567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:40.169555 18573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:40.169620 18572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:40.169679 18567 server_base.cc:1034] running on GCE node
W20260924 04:50:40.169550 18575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:40.170298 18567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:40.170938 18567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:40.172150 18567 hybrid_clock.cc:648] HybridClock initialized: now 1790225440172118 us; error 43 us; skew 500 ppm
Sep 24 04:50:40 dist-test-slave-hvcl krb5kdc[18460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:40.175325 18567 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:50:40.176265 18567 webserver.cc:460] Webserver started at http://127.0.98.1:45129/ using document root <none> and password file <none>
I20260924 04:50:40.176826 18567 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:40.176893 18567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:40.177093 18567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:40.179293 18567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6"
format_stamp: "Formatted at 2026-09-24 04:50:40 on dist-test-slave-hvcl"
server_key: "f751a1a4715d1ed9088d7d6c4393e4df"
server_key_iv: "6f61db2aef51e203da7b96cbd53a493d"
server_key_version: "encryptionkey@0"
I20260924 04:50:40.179857 18567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6"
format_stamp: "Formatted at 2026-09-24 04:50:40 on dist-test-slave-hvcl"
server_key: "f751a1a4715d1ed9088d7d6c4393e4df"
server_key_iv: "6f61db2aef51e203da7b96cbd53a493d"
server_key_version: "encryptionkey@0"
I20260924 04:50:40.183763 18567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:50:40.186308 18582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:40.187677 18567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:50:40.187795 18567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6"
format_stamp: "Formatted at 2026-09-24 04:50:40 on dist-test-slave-hvcl"
server_key: "f751a1a4715d1ed9088d7d6c4393e4df"
server_key_iv: "6f61db2aef51e203da7b96cbd53a493d"
server_key_version: "encryptionkey@0"
I20260924 04:50:40.187898 18567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:40.201694 18567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:40.204964 18567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:40.205145 18567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:40.205750 18567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:40.206565 18567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:40.206611 18567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:40.206686 18567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:40.206712 18567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:40.215205 18567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40775
I20260924 04:50:40.215255 18695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40775 every 8 connection(s)
I20260924 04:50:40.216187 18567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:50:40.219527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 18567
I20260924 04:50:40.219646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:50:40.219904 392 external_mini_cluster.cc:1428] Setting key dd7b8b8e5b7734f322a7574669b9cef5
Sep 24 04:50:40 dist-test-slave-hvcl krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:50:40.230566 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.217989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48089 (local address 127.0.98.62:34545)
0924 04:50:40.218135 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:40.218138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:40.219267 (+ 1129us) server_negotiation.cc:407] Connection header received
0924 04:50:40.220209 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:40.220212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:40.220269 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:40.220345 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:40.221667 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.222192 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.223796 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.224029 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.224117 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.226826 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:40.226830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:40.226832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:40.226857 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:40.228272 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:40.228844 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:40.228847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:40.228849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:40.228924 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:40.229233 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:40.229236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:40.229238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:40.229473 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:40.229556 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:40.230240 (+ 684us) server_negotiation.cc:299] Negotiation successful
0924 04:50:40.230391 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:50:40.231480 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.218362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34545 (local address 127.0.98.1:48089)
0924 04:50:40.219094 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:40.219151 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:40.220007 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:40.220541 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:40.220550 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:40.221022 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:40.221532 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.221546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.222308 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.222311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.223677 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.223690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.224174 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.224178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.224360 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.225006 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:50:40.225023 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:40.226650 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:40.228384 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:40.228391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:40.228408 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:40.228752 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:40.229005 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:40.229008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:40.229010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:40.229149 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:40.229682 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:40.229692 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:40.230001 (+ 309us) client_negotiation.cc:769] Sending connection context
0924 04:50:40.230190 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 04:50:40.230539 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":110,"threads_started":1}
I20260924 04:50:40.232815 18696 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34545
I20260924 04:50:40.233074 18696 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:40.233603 18696 heartbeater.cc:511] Master 127.0.98.62:34545 requested a full tablet report, sending...
I20260924 04:50:40.235252 18505 ts_manager.cc:194] Registered new tserver with Master: a1b47a34f6e74f74bf5bbe8ecde52ee6 (127.0.98.1:40775)
I20260924 04:50:40.236438 18505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:50:40.247781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34545
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:40.356956 18701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:40.357239 18701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:40.357282 18701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:40.361366 18701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:40.361445 18701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:40.361509 18701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:50:40.366950 18701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:40.368146 18701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:40.369244 18701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:50:40.379150 18701 server_base.cc:1034] running on GCE node
W20260924 04:50:40.379098 18706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:40.379098 18707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:40.379112 18709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:40.379817 18701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:40.380435 18701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:40.381613 18701 hybrid_clock.cc:648] HybridClock initialized: now 1790225440381575 us; error 52 us; skew 500 ppm
Sep 24 04:50:40 dist-test-slave-hvcl krb5kdc[18460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:40.384361 18701 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:50:40.385327 18701 webserver.cc:460] Webserver started at http://127.0.98.2:38999/ using document root <none> and password file <none>
I20260924 04:50:40.385984 18701 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:40.386046 18701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:40.386245 18701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:40.388303 18701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7"
format_stamp: "Formatted at 2026-09-24 04:50:40 on dist-test-slave-hvcl"
server_key: "b88d1341aaa7900852177f140a3f82d2"
server_key_iv: "31ab6747feea17c2c61fcec766a2a1af"
server_key_version: "encryptionkey@0"
I20260924 04:50:40.388852 18701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7"
format_stamp: "Formatted at 2026-09-24 04:50:40 on dist-test-slave-hvcl"
server_key: "b88d1341aaa7900852177f140a3f82d2"
server_key_iv: "31ab6747feea17c2c61fcec766a2a1af"
server_key_version: "encryptionkey@0"
I20260924 04:50:40.392582 18701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:50:40.395301 18716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:40.396698 18701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:50:40.396875 18701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7"
format_stamp: "Formatted at 2026-09-24 04:50:40 on dist-test-slave-hvcl"
server_key: "b88d1341aaa7900852177f140a3f82d2"
server_key_iv: "31ab6747feea17c2c61fcec766a2a1af"
server_key_version: "encryptionkey@0"
I20260924 04:50:40.396987 18701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:40.408396 18701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:40.411587 18701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:40.411782 18701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:40.412375 18701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:40.413189 18701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:40.413239 18701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:40.413292 18701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:40.413317 18701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:40.421981 18701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33159
I20260924 04:50:40.422001 18829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33159 every 8 connection(s)
I20260924 04:50:40.422955 18701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:50:40.423838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 18701
I20260924 04:50:40.423964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:50:40.424278 392 external_mini_cluster.cc:1428] Setting key 92a7396b808dba22783d553e2015a8f8
Sep 24 04:50:40 dist-test-slave-hvcl krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:50:40.443853 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.424956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36757 (local address 127.0.98.62:34545)
0924 04:50:40.425093 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:40.425097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:40.426104 (+ 1007us) server_negotiation.cc:407] Connection header received
0924 04:50:40.427223 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:40.427226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:40.427276 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:40.427346 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:40.428808 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.429463 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.430728 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.430995 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.431095 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.433772 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:40.433776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:40.433779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:40.433811 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:40.441191 (+ 7380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:40.441924 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:40.441929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:40.441931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:40.442043 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:40.442443 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:40.442447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:40.442449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:40.442705 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:40.442815 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:40.443527 (+ 712us) server_negotiation.cc:299] Negotiation successful
0924 04:50:40.443678 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:50:40.444684 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.425397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34545 (local address 127.0.98.2:36757)
0924 04:50:40.425930 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:40.425995 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:40.426996 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:40.427540 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:40.427550 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:40.428060 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:40.428604 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.428620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.429586 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.429589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.430616 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.430626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.431141 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.431145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.431315 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.431931 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:50:40.431947 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:40.433567 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:40.441400 (+ 7833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:40.441407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:40.441425 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:40.441764 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:40.442158 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:40.442161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:40.442163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:40.442345 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:40.442869 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:40.442876 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:40.443228 (+ 352us) client_negotiation.cc:769] Sending connection context
0924 04:50:40.443419 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 04:50:40.443730 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":141,"threads_started":1}
I20260924 04:50:40.446213 18830 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34545
I20260924 04:50:40.446488 18830 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:40.447063 18830 heartbeater.cc:511] Master 127.0.98.62:34545 requested a full tablet report, sending...
I20260924 04:50:40.448236 18505 ts_manager.cc:194] Registered new tserver with Master: faaa1faf3e2a4f3db4433c23ae22e2f7 (127.0.98.2:33159)
I20260924 04:50:40.448843 18505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:50:40.454409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34545
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:40.570396 18835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:40.570672 18835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:40.570721 18835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:40.574836 18835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:40.574919 18835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:40.575026 18835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:50:40.580417 18835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:40.581492 18835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:40.582501 18835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:40.591707 18843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:40.591718 18840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:40.591768 18841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:40.591858 18835 server_base.cc:1034] running on GCE node
I20260924 04:50:40.592375 18835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:40.593044 18835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:40.594238 18835 hybrid_clock.cc:648] HybridClock initialized: now 1790225440594209 us; error 45 us; skew 500 ppm
Sep 24 04:50:40 dist-test-slave-hvcl krb5kdc[18460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:40.596787 18835 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:50:40.597668 18835 webserver.cc:460] Webserver started at http://127.0.98.3:33413/ using document root <none> and password file <none>
I20260924 04:50:40.598205 18835 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:40.598271 18835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:40.598462 18835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:40.600582 18835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "70eac59d83b248c9842cf0b205fa5b14"
format_stamp: "Formatted at 2026-09-24 04:50:40 on dist-test-slave-hvcl"
server_key: "1c1aaa717e8741955c2825da2b061b68"
server_key_iv: "d7eff413759baef0423c113802059169"
server_key_version: "encryptionkey@0"
I20260924 04:50:40.601148 18835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "70eac59d83b248c9842cf0b205fa5b14"
format_stamp: "Formatted at 2026-09-24 04:50:40 on dist-test-slave-hvcl"
server_key: "1c1aaa717e8741955c2825da2b061b68"
server_key_iv: "d7eff413759baef0423c113802059169"
server_key_version: "encryptionkey@0"
I20260924 04:50:40.604904 18835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 04:50:40.607319 18850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:40.608484 18835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:50:40.608598 18835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "70eac59d83b248c9842cf0b205fa5b14"
format_stamp: "Formatted at 2026-09-24 04:50:40 on dist-test-slave-hvcl"
server_key: "1c1aaa717e8741955c2825da2b061b68"
server_key_iv: "d7eff413759baef0423c113802059169"
server_key_version: "encryptionkey@0"
I20260924 04:50:40.608695 18835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:40.618319 18835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:40.621475 18835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:40.621657 18835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:40.622238 18835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:40.623077 18835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:40.623124 18835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:40.623178 18835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:40.623204 18835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:40.631963 18835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39989
I20260924 04:50:40.631983 18963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39989 every 8 connection(s)
I20260924 04:50:40.632875 18835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:50:40.640552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 18835
I20260924 04:50:40.640681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:50:40.641000 392 external_mini_cluster.cc:1428] Setting key 3630805b54ad6bbf76020ff0012c3142
Sep 24 04:50:40 dist-test-slave-hvcl krb5kdc[18460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:50:40.646072 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.634642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40849 (local address 127.0.98.62:34545)
0924 04:50:40.634780 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:40.634784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:40.635733 (+ 949us) server_negotiation.cc:407] Connection header received
0924 04:50:40.636647 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:40.636651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:40.636709 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:40.636780 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:40.637949 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.638494 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.639703 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.639923 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.640011 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.642757 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:40.642760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:40.642762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:40.642788 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:40.644101 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:40.644595 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:40.644598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:40.644599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:40.644674 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:40.644988 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:40.644992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:40.644993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:40.645179 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:40.645261 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:40.645772 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 04:50:40.645910 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:50:40.646896 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.635140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34545 (local address 127.0.98.3:40849)
0924 04:50:40.635587 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:40.635632 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:40.636448 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:40.636967 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:40.636974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:40.637327 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:40.637800 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.637813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.638613 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.638615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.639586 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.639596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.640067 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.640071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.640244 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.640990 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:50:40.641013 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:40.642591 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:40.644212 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:40.644219 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:40.644236 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:40.644503 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:40.644754 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:40.644757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:40.644759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:40.644904 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:40.645281 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:40.645289 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:40.645548 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 04:50:40.645717 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 04:50:40.646056 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":108,"threads_started":1}
I20260924 04:50:40.648057 18964 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34545
I20260924 04:50:40.648312 18964 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:40.648866 18964 heartbeater.cc:511] Master 127.0.98.62:34545 requested a full tablet report, sending...
I20260924 04:50:40.649893 18505 ts_manager.cc:194] Registered new tserver with Master: 70eac59d83b248c9842cf0b205fa5b14 (127.0.98.3:39989)
I20260924 04:50:40.650453 18505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40849
I20260924 04:50:40.654856 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:50:40.663945 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.656701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34216 (local address 127.0.98.62:34545)
0924 04:50:40.656804 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:40.656807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:40.656969 (+ 162us) server_negotiation.cc:407] Connection header received
0924 04:50:40.657077 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:40.657079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:40.657116 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:40.657179 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:40.657892 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.658416 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.659273 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.659500 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.659585 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.661080 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:40.661083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:40.661085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:40.661109 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:40.662393 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:40.662855 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:40.662857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:40.662859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:40.662929 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:40.663207 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:40.663209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:40.663210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:40.663408 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:40.663516 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:40.663707 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 04:50:40.663785 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:50:40.667079 18505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:50:40.670104 18505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:50:40.685411 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.679307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39989 (local address 127.0.0.1:33274)
0924 04:50:40.680126 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:40.680196 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:40.680324 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:40.681280 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:40.681283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:40.681298 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:40.681377 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.681384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.682784 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.682787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.684049 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.684058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.684872 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.684875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.684982 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.685016 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 04:50:40.685121 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 04:50:40.685241 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":50,"thread_start_us":66,"threads_started":1}
I20260924 04:50:40.685566 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.679429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.0.98.1:40775)
0924 04:50:40.679986 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:40.679999 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:40.680465 (+ 466us) server_negotiation.cc:407] Connection header received
0924 04:50:40.680552 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:40.680557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:40.680797 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:40.680952 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:40.681416 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.682677 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.684042 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.684864 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.684973 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.685111 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:40.685307 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 04:50:40.685408 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":147,"threads_started":1}
I20260924 04:50:40.685877 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.680130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.0.98.2:33159)
0924 04:50:40.680568 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:40.680578 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:40.681059 (+ 481us) server_negotiation.cc:407] Connection header received
0924 04:50:40.681137 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:40.681142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:40.681327 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:40.681449 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:40.681756 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.682462 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.684153 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.684907 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.684989 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.685111 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:40.685503 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 04:50:40.685595 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":149,"threads_started":1}
I20260924 04:50:40.685887 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.679664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33159 (local address 127.0.0.1:58786)
0924 04:50:40.680886 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:40.680898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:40.680971 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:40.681570 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:40.681572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:40.681586 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:40.681644 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.681648 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.682587 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.682590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.684048 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.684058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.685174 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.685177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.685275 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.685299 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:50:40.685362 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:50:40.685428 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"mutex_wait_us":30,"thread_start_us":104,"threads_started":1}
I20260924 04:50:40.685411 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.679307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40775 (local address 127.0.0.1:42668)
0924 04:50:40.680140 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:40.680196 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:40.680325 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:40.681139 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:40.681144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:40.681165 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:40.681291 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.681297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.682813 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.682816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.683914 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.683923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.685084 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.685088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.685166 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.685184 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:50:40.685258 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:50:40.685315 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":23,"thread_start_us":98,"threads_started":1}
I20260924 04:50:40.687341 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.680030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.0.98.3:39989)
0924 04:50:40.680674 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:40.680685 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:40.680706 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:50:40.680770 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:40.680775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:40.680960 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:40.681107 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:40.681530 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.682655 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.684206 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.684760 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.684859 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.685012 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:40.685484 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 04:50:40.687193 (+ 1709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":128,"threads_started":1}
I20260924 04:50:40.688920 18764 tablet_service.cc:1467] Processing CreateTablet for tablet 9e403b0a7e0a4be7b08fac6360dd0178 (DEFAULT_TABLE table=test-table [id=d53effa7d6c54886af78ffb88e93fa36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:40.689416 18630 tablet_service.cc:1467] Processing CreateTablet for tablet 9e403b0a7e0a4be7b08fac6360dd0178 (DEFAULT_TABLE table=test-table [id=d53effa7d6c54886af78ffb88e93fa36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:40.689920 18764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e403b0a7e0a4be7b08fac6360dd0178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:40.689865 18898 tablet_service.cc:1467] Processing CreateTablet for tablet 9e403b0a7e0a4be7b08fac6360dd0178 (DEFAULT_TABLE table=test-table [id=d53effa7d6c54886af78ffb88e93fa36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:40.690490 18630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e403b0a7e0a4be7b08fac6360dd0178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:40.690948 18898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e403b0a7e0a4be7b08fac6360dd0178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:40.698472 18981 tablet_bootstrap.cc:492] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7: Bootstrap starting.
I20260924 04:50:40.698537 18982 tablet_bootstrap.cc:492] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6: Bootstrap starting.
I20260924 04:50:40.700668 18982 tablet_bootstrap.cc:654] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:40.700675 18981 tablet_bootstrap.cc:654] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:40.701244 18983 tablet_bootstrap.cc:492] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14: Bootstrap starting.
I20260924 04:50:40.701865 18981 log.cc:824] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:40.701889 18982 log.cc:824] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:40.703629 18983 tablet_bootstrap.cc:654] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:40.704162 18982 tablet_bootstrap.cc:492] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6: No bootstrap required, opened a new log
I20260924 04:50:40.704161 18981 tablet_bootstrap.cc:492] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7: No bootstrap required, opened a new log
I20260924 04:50:40.704376 18981 ts_tablet_manager.cc:1397] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:50:40.704376 18982 ts_tablet_manager.cc:1397] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:50:40.704716 18983 log.cc:824] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:40.706415 18983 tablet_bootstrap.cc:492] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14: No bootstrap required, opened a new log
I20260924 04:50:40.706598 18983 ts_tablet_manager.cc:1397] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:50:40.708343 18982 raft_consensus.cc:348] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70eac59d83b248c9842cf0b205fa5b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39989 } } peers { permanent_uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40775 } } peers { permanent_uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33159 } }
I20260924 04:50:40.708341 18981 raft_consensus.cc:348] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70eac59d83b248c9842cf0b205fa5b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39989 } } peers { permanent_uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40775 } } peers { permanent_uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33159 } }
I20260924 04:50:40.708627 18981 raft_consensus.cc:374] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:40.708627 18982 raft_consensus.cc:374] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:40.708685 18981 raft_consensus.cc:729] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faaa1faf3e2a4f3db4433c23ae22e2f7, State: Initialized, Role: FOLLOWER
I20260924 04:50:40.708688 18982 raft_consensus.cc:729] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b47a34f6e74f74bf5bbe8ecde52ee6, State: Initialized, Role: FOLLOWER
I20260924 04:50:40.709161 18981 consensus_queue.cc:260] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70eac59d83b248c9842cf0b205fa5b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39989 } } peers { permanent_uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40775 } } peers { permanent_uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33159 } }
I20260924 04:50:40.709161 18982 consensus_queue.cc:260] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70eac59d83b248c9842cf0b205fa5b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39989 } } peers { permanent_uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40775 } } peers { permanent_uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33159 } }
I20260924 04:50:40.709642 18983 raft_consensus.cc:348] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70eac59d83b248c9842cf0b205fa5b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39989 } } peers { permanent_uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40775 } } peers { permanent_uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33159 } }
I20260924 04:50:40.709896 18983 raft_consensus.cc:374] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:40.709949 18983 raft_consensus.cc:729] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70eac59d83b248c9842cf0b205fa5b14, State: Initialized, Role: FOLLOWER
I20260924 04:50:40.710222 18981 ts_tablet_manager.cc:1428] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:50:40.710222 18982 ts_tablet_manager.cc:1428] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:50:40.710422 18983 consensus_queue.cc:260] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70eac59d83b248c9842cf0b205fa5b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39989 } } peers { permanent_uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40775 } } peers { permanent_uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33159 } }
I20260924 04:50:40.710562 18696 heartbeater.cc:503] Master 127.0.98.62:34545 was elected leader, sending a full tablet report...
I20260924 04:50:40.710919 18830 heartbeater.cc:503] Master 127.0.98.62:34545 was elected leader, sending a full tablet report...
I20260924 04:50:40.711336 18983 ts_tablet_manager.cc:1428] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:50:40.711648 18964 heartbeater.cc:503] Master 127.0.98.62:34545 was elected leader, sending a full tablet report...
I20260924 04:50:40.714752 18989 raft_consensus.cc:482] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:50:40.714910 18989 raft_consensus.cc:504] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70eac59d83b248c9842cf0b205fa5b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39989 } } peers { permanent_uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40775 } } peers { permanent_uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33159 } }
I20260924 04:50:40.716076 18989 leader_election.cc:288] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1b47a34f6e74f74bf5bbe8ecde52ee6 (127.0.98.1:40775), faaa1faf3e2a4f3db4433c23ae22e2f7 (127.0.98.2:33159)
W20260924 04:50:40.717334 18697 tablet.cc:2369] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:50:40.719972 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.716500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47769 (local address 127.0.98.2:33159)
0924 04:50:40.716621 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:40.716626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:40.716825 (+ 199us) server_negotiation.cc:407] Connection header received
0924 04:50:40.716867 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:40.716870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:40.716903 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:40.716958 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:40.717205 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.717763 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.718772 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.719331 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.719427 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.719492 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:40.719790 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 04:50:40.719859 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:50:40.719985 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.716293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33159 (local address 127.0.98.3:47769)
0924 04:50:40.716619 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:40.716633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:40.716743 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:40.716981 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:40.716984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:40.716998 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:40.717112 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.717117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.717858 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.717861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.718653 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.718661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.719550 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.719553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.719679 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.719697 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:50:40.719762 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:50:40.719845 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":24,"spinlock_wait_cycles":32512,"thread_start_us":102,"threads_started":1}
I20260924 04:50:40.720114 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.716500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47771 (local address 127.0.98.1:40775)
0924 04:50:40.716730 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:40.716734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:40.716746 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:50:40.716823 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:40.716826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:40.716866 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:40.716926 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:40.717184 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.717868 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.718892 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.719420 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.719514 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.719574 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:40.719951 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 04:50:40.720016 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260924 04:50:40.720124 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.716273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40775 (local address 127.0.98.3:47771)
0924 04:50:40.716618 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:40.716632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:40.716745 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:40.716941 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:40.716944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:40.716959 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:40.717092 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.717098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.717967 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.717971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.718797 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:40.718807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.719634 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.719637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:40.719838 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.719856 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:50:40.719922 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:50:40.719990 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":21,"spinlock_wait_cycles":5888}
I20260924 04:50:40.720623 18784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e403b0a7e0a4be7b08fac6360dd0178" candidate_uuid: "70eac59d83b248c9842cf0b205fa5b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7" is_pre_election: true
I20260924 04:50:40.720788 18650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e403b0a7e0a4be7b08fac6360dd0178" candidate_uuid: "70eac59d83b248c9842cf0b205fa5b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6" is_pre_election: true
I20260924 04:50:40.720906 18784 raft_consensus.cc:2454] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70eac59d83b248c9842cf0b205fa5b14 in term 0.
I20260924 04:50:40.721050 18650 raft_consensus.cc:2454] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70eac59d83b248c9842cf0b205fa5b14 in term 0.
I20260924 04:50:40.721344 18852 leader_election.cc:302] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70eac59d83b248c9842cf0b205fa5b14, faaa1faf3e2a4f3db4433c23ae22e2f7; no voters:
I20260924 04:50:40.721576 18989 raft_consensus.cc:2781] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:50:40.721654 18989 raft_consensus.cc:482] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:50:40.721701 18989 raft_consensus.cc:3037] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:40.722695 18989 raft_consensus.cc:504] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70eac59d83b248c9842cf0b205fa5b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39989 } } peers { permanent_uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40775 } } peers { permanent_uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33159 } }
I20260924 04:50:40.723040 18989 leader_election.cc:288] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [CANDIDATE]: Term 1 election: Requested vote from peers a1b47a34f6e74f74bf5bbe8ecde52ee6 (127.0.98.1:40775), faaa1faf3e2a4f3db4433c23ae22e2f7 (127.0.98.2:33159)
I20260924 04:50:40.723344 18650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e403b0a7e0a4be7b08fac6360dd0178" candidate_uuid: "70eac59d83b248c9842cf0b205fa5b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6"
I20260924 04:50:40.723366 18784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e403b0a7e0a4be7b08fac6360dd0178" candidate_uuid: "70eac59d83b248c9842cf0b205fa5b14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7"
I20260924 04:50:40.723470 18650 raft_consensus.cc:3037] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:40.723470 18784 raft_consensus.cc:3037] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:40.724704 18650 raft_consensus.cc:2454] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70eac59d83b248c9842cf0b205fa5b14 in term 1.
I20260924 04:50:40.724792 18784 raft_consensus.cc:2454] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70eac59d83b248c9842cf0b205fa5b14 in term 1.
I20260924 04:50:40.725072 18853 leader_election.cc:302] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70eac59d83b248c9842cf0b205fa5b14, a1b47a34f6e74f74bf5bbe8ecde52ee6; no voters:
I20260924 04:50:40.725253 18989 raft_consensus.cc:2781] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:40.725494 18989 raft_consensus.cc:686] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [term 1 LEADER]: Becoming Leader. State: Replica: 70eac59d83b248c9842cf0b205fa5b14, State: Running, Role: LEADER
I20260924 04:50:40.725841 18989 consensus_queue.cc:237] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70eac59d83b248c9842cf0b205fa5b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39989 } } peers { permanent_uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40775 } } peers { permanent_uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33159 } }
I20260924 04:50:40.729351 18503 catalog_manager.cc:5495] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70eac59d83b248c9842cf0b205fa5b14 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "70eac59d83b248c9842cf0b205fa5b14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70eac59d83b248c9842cf0b205fa5b14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33159 } health_report { overall_health: UNKNOWN } } }
I20260924 04:50:40.751030 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.747631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.0.98.3:39989)
0924 04:50:40.747761 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:40.747766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:40.747781 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:50:40.747855 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:40.747858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:40.747918 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:40.748008 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:50:40.748254 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.748763 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.749589 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:40.749802 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:40.749885 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:40.750180 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:50:40.750585 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:50:40.750714 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:40.750820 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 04:50:40.750862 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:50:40.756685 18784 raft_consensus.cc:1261] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7 [term 1 FOLLOWER]: Refusing update from remote peer 70eac59d83b248c9842cf0b205fa5b14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:50:40.756685 18650 raft_consensus.cc:1261] T 9e403b0a7e0a4be7b08fac6360dd0178 P a1b47a34f6e74f74bf5bbe8ecde52ee6 [term 1 FOLLOWER]: Refusing update from remote peer 70eac59d83b248c9842cf0b205fa5b14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:50:40.757336 18991 consensus_queue.cc:1035] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faaa1faf3e2a4f3db4433c23ae22e2f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:50:40.757516 18989 consensus_queue.cc:1035] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1b47a34f6e74f74bf5bbe8ecde52ee6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:50:40.764448 18994 mvcc.cc:205] Tried to move back new op lower bound from 7332763405332463616 to 7332763405215666176. Current Snapshot: MvccSnapshot[applied={T|T < 7332763405332463616}]
I20260924 04:50:40.764875 18996 mvcc.cc:205] Tried to move back new op lower bound from 7332763405332463616 to 7332763405215666176. Current Snapshot: MvccSnapshot[applied={T|T < 7332763405332463616}]
I20260924 04:50:40.767263 18995 mvcc.cc:205] Tried to move back new op lower bound from 7332763405332463616 to 7332763405215666176. Current Snapshot: MvccSnapshot[applied={T|T < 7332763405332463616}]
I20260924 04:50:40.773072 18505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:50:40.773244 18505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:50:40.775700 18505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5412813351584d05be3ca598febd5c47: Sending DeleteTablet for 3 replicas of tablet 9e403b0a7e0a4be7b08fac6360dd0178
I20260924 04:50:40.776516 18630 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e403b0a7e0a4be7b08fac6360dd0178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42668
I20260924 04:50:40.776671 18764 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e403b0a7e0a4be7b08fac6360dd0178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58786
I20260924 04:50:40.777176 19004 tablet_replica.cc:330] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7: stopping tablet replica
I20260924 04:50:40.777165 18898 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e403b0a7e0a4be7b08fac6360dd0178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33274
I20260924 04:50:40.777344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 18567
I20260924 04:50:40.777516 19004 raft_consensus.cc:2229] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:50:40.777684 19005 tablet_replica.cc:330] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14: stopping tablet replica
I20260924 04:50:40.777853 19004 raft_consensus.cc:2258] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:50:40.778009 19005 raft_consensus.cc:2229] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:50:40.778460 19005 raft_consensus.cc:2258] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:50:40.779460 19004 ts_tablet_manager.cc:1905] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:50:40.779919 19005 ts_tablet_manager.cc:1905] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:50:40.782220 19004 ts_tablet_manager.cc:1918] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:50:40.782317 19004 log.cc:1196] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/9e403b0a7e0a4be7b08fac6360dd0178
I20260924 04:50:40.782436 19005 ts_tablet_manager.cc:1918] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:50:40.782516 19005 log.cc:1196] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/9e403b0a7e0a4be7b08fac6360dd0178
I20260924 04:50:40.782625 19004 ts_tablet_manager.cc:1939] T 9e403b0a7e0a4be7b08fac6360dd0178 P faaa1faf3e2a4f3db4433c23ae22e2f7: Deleting consensus metadata
I20260924 04:50:40.782799 19005 ts_tablet_manager.cc:1939] T 9e403b0a7e0a4be7b08fac6360dd0178 P 70eac59d83b248c9842cf0b205fa5b14: Deleting consensus metadata
I20260924 04:50:40.783829 18489 catalog_manager.cc:4841] TS 70eac59d83b248c9842cf0b205fa5b14 (127.0.98.3:39989): tablet 9e403b0a7e0a4be7b08fac6360dd0178 (table test-table [id=d53effa7d6c54886af78ffb88e93fa36]) successfully deleted
I20260924 04:50:40.784042 18490 catalog_manager.cc:4841] TS faaa1faf3e2a4f3db4433c23ae22e2f7 (127.0.98.2:33159): tablet 9e403b0a7e0a4be7b08fac6360dd0178 (table test-table [id=d53effa7d6c54886af78ffb88e93fa36]) successfully deleted
W20260924 04:50:40.788647 18491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40775 (error 108)
I20260924 04:50:40.788798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 18701
I20260924 04:50:40.790247 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:40.789948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40775 (local address 127.0.0.1:42672)
0924 04:50:40.790076 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:40.790145 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 04:50:40.791114 18491 catalog_manager.cc:4584] TS a1b47a34f6e74f74bf5bbe8ecde52ee6 (127.0.98.1:40775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e403b0a7e0a4be7b08fac6360dd0178: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40775: connect: Connection refused (error 111)
I20260924 04:50:40.794014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 18835
I20260924 04:50:40.798560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 18473
2026-09-24T04:50:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2155 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T04:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:40Z Disabled control of system clock
I20260924 04:50:40.826434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42015
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:50:40.935277 19010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:40.935565 19010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:40.935604 19010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:40.939673 19010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:40.939735 19010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:40.939749 19010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:40.939765 19010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:40.939779 19010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:40.944698 19010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:40.945806 19010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:40.947324 19010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:40.955655 19018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:40.955675 19016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:40.955773 19010 server_base.cc:1034] running on GCE node
W20260924 04:50:40.955675 19015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:40.956545 19010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:40.957739 19010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:40.958937 19010 hybrid_clock.cc:648] HybridClock initialized: now 1790225440958921 us; error 34 us; skew 500 ppm
I20260924 04:50:40.960956 19010 webserver.cc:460] Webserver started at http://127.0.98.62:43027/ using document root <none> and password file <none>
I20260924 04:50:40.961545 19010 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:40.961616 19010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:40.961814 19010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:40.963954 19010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "d1bcbff63d724b8ab2cba4807f949811"
format_stamp: "Formatted at 2026-09-24 04:50:40 on dist-test-slave-hvcl"
server_key: "92123523f4e5aadca5fb1285a1a830a3"
server_key_iv: "caa9a413a8625562c9bbe6c37e98fbfa"
server_key_version: "encryptionkey@0"
I20260924 04:50:40.964511 19010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "d1bcbff63d724b8ab2cba4807f949811"
format_stamp: "Formatted at 2026-09-24 04:50:40 on dist-test-slave-hvcl"
server_key: "92123523f4e5aadca5fb1285a1a830a3"
server_key_iv: "caa9a413a8625562c9bbe6c37e98fbfa"
server_key_version: "encryptionkey@0"
I20260924 04:50:40.968461 19010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:50:40.970949 19024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:40.972329 19010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:50:40.972450 19010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "d1bcbff63d724b8ab2cba4807f949811"
format_stamp: "Formatted at 2026-09-24 04:50:40 on dist-test-slave-hvcl"
server_key: "92123523f4e5aadca5fb1285a1a830a3"
server_key_iv: "caa9a413a8625562c9bbe6c37e98fbfa"
server_key_version: "encryptionkey@0"
I20260924 04:50:40.972563 19010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:40.985221 19010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:40.985972 19010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:40.986155 19010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:40.992774 19010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42015
I20260924 04:50:40.992821 19076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42015 every 8 connection(s)
I20260924 04:50:40.993749 19010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:40.996971 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:41.002209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 19010
I20260924 04:50:41.002339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:41.002625 392 external_mini_cluster.cc:1428] Setting key b8381f09decf80f68fd138af8b821a89
I20260924 04:50:41.002738 19077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811: Bootstrap starting.
I20260924 04:50:41.005285 19077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:41.006024 19077 log.cc:824] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:41.008212 19077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811: No bootstrap required, opened a new log
I20260924 04:50:41.012096 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.004061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.0.98.62:42015)
0924 04:50:41.004710 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:41.004724 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:41.004759 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:50:41.005434 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:41.005453 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:41.005873 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:41.006172 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:41.006573 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.007649 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.008908 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.009394 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.009514 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.010338 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:41.010343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:41.010373 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:41.010746 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:41.010799 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:41.010819 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:41.010892 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:41.011057 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 04:50:41.011318 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":119,"threads_started":1}
I20260924 04:50:41.012259 19077 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bcbff63d724b8ab2cba4807f949811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } }
I20260924 04:50:41.012511 19077 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:41.012578 19077 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bcbff63d724b8ab2cba4807f949811, State: Initialized, Role: FOLLOWER
I20260924 04:50:41.012987 19077 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bcbff63d724b8ab2cba4807f949811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } }
I20260924 04:50:41.013125 19077 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:41.013198 19077 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:41.013290 19077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:41.014473 19077 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bcbff63d724b8ab2cba4807f949811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } }
I20260924 04:50:41.014794 19077 leader_election.cc:302] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1bcbff63d724b8ab2cba4807f949811; no voters:
I20260924 04:50:41.015087 19077 leader_election.cc:288] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:41.015193 19082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:41.015373 19082 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [term 1 LEADER]: Becoming Leader. State: Replica: d1bcbff63d724b8ab2cba4807f949811, State: Running, Role: LEADER
I20260924 04:50:41.015677 19082 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bcbff63d724b8ab2cba4807f949811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } }
I20260924 04:50:41.016371 19077 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:41.017379 19083 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1bcbff63d724b8ab2cba4807f949811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bcbff63d724b8ab2cba4807f949811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } } }
I20260924 04:50:41.017509 19083 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:41.017779 19084 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1bcbff63d724b8ab2cba4807f949811. Latest consensus state: current_term: 1 leader_uuid: "d1bcbff63d724b8ab2cba4807f949811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bcbff63d724b8ab2cba4807f949811" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } } }
I20260924 04:50:41.017871 19091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:41.017874 19084 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:41.021163 19091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:41.026827 19091 catalog_manager.cc:1347] Generated new cluster ID: 8fd70f45bc9f47efbd21226935e056fe
I20260924 04:50:41.026919 19091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:41.057767 19091 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:41.058708 19091 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:41.064227 19091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811: Generated new TSK 0
I20260924 04:50:41.065137 19091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:50:41.073573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42015
--builtin_ntp_servers=127.0.98.20:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:50:41.201498 19101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:41.201782 19101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:41.201822 19101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:41.205873 19101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:41.205942 19101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:41.206002 19101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:50:41.210817 19101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:41.211971 19101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:41.213380 19101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:41.223316 19106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:41.223342 19109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:41.223380 19101 server_base.cc:1034] running on GCE node
W20260924 04:50:41.223327 19107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:41.223994 19101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:41.224696 19101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:41.225884 19101 hybrid_clock.cc:648] HybridClock initialized: now 1790225441225849 us; error 47 us; skew 500 ppm
I20260924 04:50:41.227869 19101 webserver.cc:460] Webserver started at http://127.0.98.1:35733/ using document root <none> and password file <none>
I20260924 04:50:41.228477 19101 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:41.228546 19101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:41.228744 19101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:41.230857 19101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc4cec4a1e0d4215b620620adcfc0705"
format_stamp: "Formatted at 2026-09-24 04:50:41 on dist-test-slave-hvcl"
server_key: "fbbd3eb016f060a0f63448d12e4a0ae5"
server_key_iv: "745c0c1bc5f5e02e49559facf938aee5"
server_key_version: "encryptionkey@0"
I20260924 04:50:41.231469 19101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc4cec4a1e0d4215b620620adcfc0705"
format_stamp: "Formatted at 2026-09-24 04:50:41 on dist-test-slave-hvcl"
server_key: "fbbd3eb016f060a0f63448d12e4a0ae5"
server_key_iv: "745c0c1bc5f5e02e49559facf938aee5"
server_key_version: "encryptionkey@0"
I20260924 04:50:41.235359 19101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:50:41.238138 19115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:41.239601 19101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:50:41.239775 19101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "cc4cec4a1e0d4215b620620adcfc0705"
format_stamp: "Formatted at 2026-09-24 04:50:41 on dist-test-slave-hvcl"
server_key: "fbbd3eb016f060a0f63448d12e4a0ae5"
server_key_iv: "745c0c1bc5f5e02e49559facf938aee5"
server_key_version: "encryptionkey@0"
I20260924 04:50:41.239902 19101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:41.251417 19101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:41.252099 19101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:41.252267 19101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:41.252847 19101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:41.253834 19101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:41.253881 19101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:41.253923 19101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:41.253952 19101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:41.261989 19101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46747
I20260924 04:50:41.262014 19228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46747 every 8 connection(s)
I20260924 04:50:41.262943 19101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:50:41.270687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 19101
I20260924 04:50:41.270906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:50:41.271250 392 external_mini_cluster.cc:1428] Setting key d197149a3cda4a8adc1e62fb046020cf
I20260924 04:50:41.272414 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.264884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41093 (local address 127.0.98.62:42015)
0924 04:50:41.265002 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:41.265006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:41.266041 (+ 1035us) server_negotiation.cc:407] Connection header received
0924 04:50:41.267427 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:41.267430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:41.267480 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:41.267554 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:41.268444 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.268970 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.270244 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.270494 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.270582 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.271918 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:41.271922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:41.271924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:41.271938 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:41.271957 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:41.271966 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:41.272031 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:41.272187 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 04:50:41.272246 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:50:41.273312 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.265264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42015 (local address 127.0.98.1:41093)
0924 04:50:41.265886 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:41.265926 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:41.267204 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:41.267702 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:41.267709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:41.267795 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:41.268325 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.268339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.269106 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.269109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.270073 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.270084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.270676 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.270680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.270924 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.271613 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:50:41.271615 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:41.271675 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:50:41.271749 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:41.272050 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:41.272057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:41.272062 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:50:41.272170 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 04:50:41.272416 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":105,"threads_started":1}
I20260924 04:50:41.273311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42015
--builtin_ntp_servers=127.0.98.20:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:50:41.274446 19229 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42015
I20260924 04:50:41.274669 19229 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:41.275225 19229 heartbeater.cc:511] Master 127.0.98.62:42015 requested a full tablet report, sending...
I20260924 04:50:41.276899 19041 ts_manager.cc:194] Registered new tserver with Master: cc4cec4a1e0d4215b620620adcfc0705 (127.0.98.1:46747)
I20260924 04:50:41.278533 19041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41093
W20260924 04:50:41.382081 19232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:41.382354 19232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:41.382391 19232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:41.386312 19232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:41.386364 19232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:41.386417 19232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:50:41.391026 19232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:41.392050 19232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:41.393338 19232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:41.402819 19237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:41.402832 19240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:41.402871 19232 server_base.cc:1034] running on GCE node
W20260924 04:50:41.402832 19238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:41.403558 19232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:41.404197 19232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:41.405395 19232 hybrid_clock.cc:648] HybridClock initialized: now 1790225441405370 us; error 37 us; skew 500 ppm
I20260924 04:50:41.407402 19232 webserver.cc:460] Webserver started at http://127.0.98.2:35575/ using document root <none> and password file <none>
I20260924 04:50:41.408041 19232 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:41.408109 19232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:41.408313 19232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:41.410460 19232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "b35df3f445154bae9d7a80a6f4a18a34"
format_stamp: "Formatted at 2026-09-24 04:50:41 on dist-test-slave-hvcl"
server_key: "82838e6d14f0636be3b18b7cedf61429"
server_key_iv: "ff66a86af496b953710b9cef1dae244b"
server_key_version: "encryptionkey@0"
I20260924 04:50:41.411063 19232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b35df3f445154bae9d7a80a6f4a18a34"
format_stamp: "Formatted at 2026-09-24 04:50:41 on dist-test-slave-hvcl"
server_key: "82838e6d14f0636be3b18b7cedf61429"
server_key_iv: "ff66a86af496b953710b9cef1dae244b"
server_key_version: "encryptionkey@0"
I20260924 04:50:41.414898 19232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 04:50:41.417685 19246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:41.419127 19232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 04:50:41.419258 19232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "b35df3f445154bae9d7a80a6f4a18a34"
format_stamp: "Formatted at 2026-09-24 04:50:41 on dist-test-slave-hvcl"
server_key: "82838e6d14f0636be3b18b7cedf61429"
server_key_iv: "ff66a86af496b953710b9cef1dae244b"
server_key_version: "encryptionkey@0"
I20260924 04:50:41.419386 19232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:41.457860 19232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:41.458585 19232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:41.458765 19232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:41.459412 19232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:41.460377 19232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:41.460433 19232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:41.460486 19232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:41.460511 19232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:41.469049 19232 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46629
I20260924 04:50:41.469084 19359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46629 every 8 connection(s)
I20260924 04:50:41.469995 19232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:50:41.479060 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.471873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52247 (local address 127.0.98.62:42015)
0924 04:50:41.472017 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:41.472021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:41.473090 (+ 1069us) server_negotiation.cc:407] Connection header received
0924 04:50:41.474471 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:41.474474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:41.474533 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:41.474605 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:41.475574 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.476148 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.477338 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.477545 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.477619 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.478620 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:41.478623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:41.478625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:41.478638 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:41.478658 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:41.478665 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:41.478718 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:41.478827 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 04:50:41.478901 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:50:41.479837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 19232
I20260924 04:50:41.479944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:50:41.479959 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.472284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42015 (local address 127.0.98.2:52247)
0924 04:50:41.472940 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:41.472982 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:41.474248 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:41.474795 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:41.474803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:41.474895 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:41.475435 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.475449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.476272 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.476275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.477204 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.477229 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.477664 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.477667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.477812 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.478372 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:50:41.478374 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:41.478415 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:50:41.478482 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:41.478725 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:41.478730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:41.478734 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:50:41.478824 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:50:41.479095 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260924 04:50:41.480240 392 external_mini_cluster.cc:1428] Setting key a8a9a4473eda4941c99ba156c7dc3e03
I20260924 04:50:41.481180 19360 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42015
I20260924 04:50:41.481437 19360 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:41.482015 19360 heartbeater.cc:511] Master 127.0.98.62:42015 requested a full tablet report, sending...
I20260924 04:50:41.482098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42015
--builtin_ntp_servers=127.0.98.20:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:50:41.483207 19041 ts_manager.cc:194] Registered new tserver with Master: b35df3f445154bae9d7a80a6f4a18a34 (127.0.98.2:46629)
I20260924 04:50:41.483799 19041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52247
W20260924 04:50:41.595872 19363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:41.596160 19363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:41.596262 19363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:41.600498 19363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:41.600564 19363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:41.600646 19363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:50:41.605607 19363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:41.606825 19363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:41.608383 19363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:50:41.618224 19363 server_base.cc:1034] running on GCE node
W20260924 04:50:41.618162 19368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:41.618162 19371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:41.618177 19369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:41.618997 19363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:41.619663 19363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:41.620822 19363 hybrid_clock.cc:648] HybridClock initialized: now 1790225441620792 us; error 45 us; skew 500 ppm
I20260924 04:50:41.622819 19363 webserver.cc:460] Webserver started at http://127.0.98.3:40687/ using document root <none> and password file <none>
I20260924 04:50:41.623494 19363 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:41.623569 19363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:41.623772 19363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:41.625875 19363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "92ecf33d9c1849b59e455655bcfed662"
format_stamp: "Formatted at 2026-09-24 04:50:41 on dist-test-slave-hvcl"
server_key: "f38dc39a100297360174f97c5c4b18ed"
server_key_iv: "2d197eed21f6dfcadd24caba63311b89"
server_key_version: "encryptionkey@0"
I20260924 04:50:41.626386 19363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "92ecf33d9c1849b59e455655bcfed662"
format_stamp: "Formatted at 2026-09-24 04:50:41 on dist-test-slave-hvcl"
server_key: "f38dc39a100297360174f97c5c4b18ed"
server_key_iv: "2d197eed21f6dfcadd24caba63311b89"
server_key_version: "encryptionkey@0"
I20260924 04:50:41.630164 19363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:50:41.632643 19377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:41.633952 19363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:50:41.634068 19363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "92ecf33d9c1849b59e455655bcfed662"
format_stamp: "Formatted at 2026-09-24 04:50:41 on dist-test-slave-hvcl"
server_key: "f38dc39a100297360174f97c5c4b18ed"
server_key_iv: "2d197eed21f6dfcadd24caba63311b89"
server_key_version: "encryptionkey@0"
I20260924 04:50:41.634166 19363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:41.671355 19363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:41.672080 19363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:41.672241 19363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:41.672801 19363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:41.673734 19363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:41.673794 19363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:41.673835 19363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:41.673849 19363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:41.682408 19363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34863
I20260924 04:50:41.682509 19490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34863 every 8 connection(s)
I20260924 04:50:41.683393 19363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:50:41.688848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 19363
I20260924 04:50:41.688979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:50:41.689282 392 external_mini_cluster.cc:1428] Setting key d9a7e9b03a28bd1c2b5ed356766132c7
I20260924 04:50:41.693014 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.685303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36853 (local address 127.0.98.62:42015)
0924 04:50:41.685453 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:41.685456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:41.686510 (+ 1054us) server_negotiation.cc:407] Connection header received
0924 04:50:41.687966 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:41.687969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:41.688029 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:41.688101 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:41.689234 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.689860 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.691212 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.691453 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.691545 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.692571 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:41.692574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:41.692577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:41.692591 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:41.692610 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:41.692618 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:41.692684 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:41.692811 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 04:50:41.692872 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:50:41.693913 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.685705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42015 (local address 127.0.98.3:36853)
0924 04:50:41.686362 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:41.686404 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:41.687732 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:41.688290 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:41.688298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:41.688387 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:41.689066 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.689087 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.689979 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.689984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.691088 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.691099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.691586 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.691590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.691764 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.692321 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:50:41.692322 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:41.692363 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:50:41.692431 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:41.692684 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:41.692693 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:41.692698 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:50:41.692808 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 04:50:41.693100 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":112,"threads_started":1}
I20260924 04:50:41.695153 19491 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42015
I20260924 04:50:41.695412 19491 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:41.695971 19491 heartbeater.cc:511] Master 127.0.98.62:42015 requested a full tablet report, sending...
I20260924 04:50:41.697008 19041 ts_manager.cc:194] Registered new tserver with Master: 92ecf33d9c1849b59e455655bcfed662 (127.0.98.3:34863)
I20260924 04:50:41.697559 19041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36853
I20260924 04:50:41.703428 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:50:41.709273 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.705399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.0.98.62:42015)
0924 04:50:41.705533 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:41.705538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:41.705659 (+ 121us) server_negotiation.cc:407] Connection header received
0924 04:50:41.705841 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:41.705844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:41.705888 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:41.705952 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:41.706247 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.706762 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.707613 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.707825 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.707910 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.708747 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:41.708750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:41.708753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:41.708768 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:41.708788 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:41.708796 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:41.708902 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:41.709060 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 04:50:41.709104 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:50:41.712354 19041 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:50:41.715438 19041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:50:41.730099 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.724967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46747 (local address 127.0.0.1:38422)
0924 04:50:41.725368 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:41.725381 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:41.725454 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:41.726032 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:41.726035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:41.726071 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:41.726205 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.726212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.727730 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.727733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.728814 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.728823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.729595 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.729599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.729671 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.729736 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 04:50:41.729846 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 04:50:41.729976 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":73,"spinlock_wait_cycles":1792,"thread_start_us":139,"threads_started":1}
I20260924 04:50:41.730157 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.725115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.0.98.1:46747)
0924 04:50:41.725537 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:41.725549 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:41.725574 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:50:41.725653 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:41.725659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:41.725888 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:41.726334 (+ 446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:41.726461 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.727609 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.728919 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.729451 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.729582 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.729778 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:41.729890 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 04:50:41.729996 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":122,"threads_started":1}
I20260924 04:50:41.730512 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.724292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34863 (local address 127.0.0.1:49022)
0924 04:50:41.724694 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:41.724742 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:41.724878 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:41.726029 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:41.726034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:41.726068 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:41.726203 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.726210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.727254 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.727257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.728479 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.728488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.729530 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.729534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.729657 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.730246 (+ 589us) client_negotiation.cc:769] Sending connection context
0924 04:50:41.730316 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:50:41.730388 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":589,"spinlock_wait_cycles":35712,"thread_start_us":96,"threads_started":1}
I20260924 04:50:41.730801 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.724189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46629 (local address 127.0.0.1:53300)
0924 04:50:41.724695 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:41.724742 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:41.724878 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:41.726216 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:41.726219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:41.726235 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:41.726305 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.726311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.728246 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.728249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.729335 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.729345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.730426 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.730430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.730529 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.730550 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:50:41.730605 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 04:50:41.730673 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":2,"spinlock_wait_cycles":3456,"thread_start_us":87,"threads_started":1}
I20260924 04:50:41.730892 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.724291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.0.98.2:46629)
0924 04:50:41.725790 (+ 1499us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:41.725800 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:41.725819 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:50:41.725893 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:41.725897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:41.726055 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:41.726225 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:41.726399 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.727609 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.729467 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.730205 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.730411 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.730587 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:41.730644 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 04:50:41.730737 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1401,"thread_start_us":107,"threads_started":1}
I20260924 04:50:41.730914 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.725226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.0.98.3:34863)
0924 04:50:41.725660 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:41.725671 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:41.725692 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:50:41.725753 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:41.725757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:41.725893 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:41.726027 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:41.726331 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.727151 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.728612 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.729414 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.729722 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.729903 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:41.730651 (+ 748us) server_negotiation.cc:299] Negotiation successful
0924 04:50:41.730767 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":154,"threads_started":1}
I20260924 04:50:41.733636 19294 tablet_service.cc:1467] Processing CreateTablet for tablet b47124f69e6a475994cd8f6455e309ac (DEFAULT_TABLE table=test-table [id=64475697b00442d28033e92e38a82b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:41.733636 19163 tablet_service.cc:1467] Processing CreateTablet for tablet b47124f69e6a475994cd8f6455e309ac (DEFAULT_TABLE table=test-table [id=64475697b00442d28033e92e38a82b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:41.733871 19425 tablet_service.cc:1467] Processing CreateTablet for tablet b47124f69e6a475994cd8f6455e309ac (DEFAULT_TABLE table=test-table [id=64475697b00442d28033e92e38a82b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:41.734737 19425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47124f69e6a475994cd8f6455e309ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:41.734742 19294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47124f69e6a475994cd8f6455e309ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:41.734742 19163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47124f69e6a475994cd8f6455e309ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:41.742069 19510 tablet_bootstrap.cc:492] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34: Bootstrap starting.
I20260924 04:50:41.742514 19508 tablet_bootstrap.cc:492] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662: Bootstrap starting.
I20260924 04:50:41.743273 19509 tablet_bootstrap.cc:492] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705: Bootstrap starting.
I20260924 04:50:41.744314 19510 tablet_bootstrap.cc:654] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:41.744359 19508 tablet_bootstrap.cc:654] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:41.745079 19508 log.cc:824] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:41.745137 19510 log.cc:824] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:41.745333 19509 tablet_bootstrap.cc:654] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:41.746124 19509 log.cc:824] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:41.747284 19510 tablet_bootstrap.cc:492] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34: No bootstrap required, opened a new log
I20260924 04:50:41.747491 19508 tablet_bootstrap.cc:492] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662: No bootstrap required, opened a new log
I20260924 04:50:41.747495 19510 ts_tablet_manager.cc:1397] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:50:41.747673 19508 ts_tablet_manager.cc:1397] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:50:41.748576 19509 tablet_bootstrap.cc:492] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705: No bootstrap required, opened a new log
I20260924 04:50:41.748760 19509 ts_tablet_manager.cc:1397] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:50:41.750841 19510 raft_consensus.cc:348] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35df3f445154bae9d7a80a6f4a18a34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46629 } } peers { permanent_uuid: "92ecf33d9c1849b59e455655bcfed662" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34863 } } peers { permanent_uuid: "cc4cec4a1e0d4215b620620adcfc0705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46747 } }
I20260924 04:50:41.751143 19510 raft_consensus.cc:374] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:41.751197 19510 raft_consensus.cc:729] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35df3f445154bae9d7a80a6f4a18a34, State: Initialized, Role: FOLLOWER
I20260924 04:50:41.751675 19510 consensus_queue.cc:260] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35df3f445154bae9d7a80a6f4a18a34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46629 } } peers { permanent_uuid: "92ecf33d9c1849b59e455655bcfed662" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34863 } } peers { permanent_uuid: "cc4cec4a1e0d4215b620620adcfc0705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46747 } }
I20260924 04:50:41.751921 19508 raft_consensus.cc:348] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35df3f445154bae9d7a80a6f4a18a34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46629 } } peers { permanent_uuid: "92ecf33d9c1849b59e455655bcfed662" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34863 } } peers { permanent_uuid: "cc4cec4a1e0d4215b620620adcfc0705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46747 } }
I20260924 04:50:41.752180 19508 raft_consensus.cc:374] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:41.752239 19508 raft_consensus.cc:729] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecf33d9c1849b59e455655bcfed662, State: Initialized, Role: FOLLOWER
I20260924 04:50:41.752506 19360 heartbeater.cc:503] Master 127.0.98.62:42015 was elected leader, sending a full tablet report...
I20260924 04:50:41.752699 19510 ts_tablet_manager.cc:1428] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:50:41.752873 19509 raft_consensus.cc:348] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35df3f445154bae9d7a80a6f4a18a34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46629 } } peers { permanent_uuid: "92ecf33d9c1849b59e455655bcfed662" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34863 } } peers { permanent_uuid: "cc4cec4a1e0d4215b620620adcfc0705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46747 } }
I20260924 04:50:41.753099 19509 raft_consensus.cc:374] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:41.753141 19509 raft_consensus.cc:729] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4cec4a1e0d4215b620620adcfc0705, State: Initialized, Role: FOLLOWER
I20260924 04:50:41.753134 19508 consensus_queue.cc:260] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35df3f445154bae9d7a80a6f4a18a34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46629 } } peers { permanent_uuid: "92ecf33d9c1849b59e455655bcfed662" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34863 } } peers { permanent_uuid: "cc4cec4a1e0d4215b620620adcfc0705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46747 } }
I20260924 04:50:41.753527 19509 consensus_queue.cc:260] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35df3f445154bae9d7a80a6f4a18a34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46629 } } peers { permanent_uuid: "92ecf33d9c1849b59e455655bcfed662" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34863 } } peers { permanent_uuid: "cc4cec4a1e0d4215b620620adcfc0705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46747 } }
I20260924 04:50:41.754060 19508 ts_tablet_manager.cc:1428] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:50:41.754173 19229 heartbeater.cc:503] Master 127.0.98.62:42015 was elected leader, sending a full tablet report...
I20260924 04:50:41.754365 19491 heartbeater.cc:503] Master 127.0.98.62:42015 was elected leader, sending a full tablet report...
I20260924 04:50:41.754458 19509 ts_tablet_manager.cc:1428] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 04:50:41.764627 19230 tablet.cc:2369] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:50:41.877121 19516 raft_consensus.cc:482] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:50:41.877363 19516 raft_consensus.cc:504] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35df3f445154bae9d7a80a6f4a18a34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46629 } } peers { permanent_uuid: "92ecf33d9c1849b59e455655bcfed662" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34863 } } peers { permanent_uuid: "cc4cec4a1e0d4215b620620adcfc0705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46747 } }
I20260924 04:50:41.878634 19516 leader_election.cc:288] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35df3f445154bae9d7a80a6f4a18a34 (127.0.98.2:46629), 92ecf33d9c1849b59e455655bcfed662 (127.0.98.3:34863)
I20260924 04:50:41.882690 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.879031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34491 (local address 127.0.98.2:46629)
0924 04:50:41.879156 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:41.879160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:41.879325 (+ 165us) server_negotiation.cc:407] Connection header received
0924 04:50:41.879504 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:41.879507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:41.879561 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:41.879630 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:41.879936 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.880561 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.881726 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.882101 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.882188 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.882248 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:41.882490 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 04:50:41.882557 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:50:41.882694 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.879030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52671 (local address 127.0.98.3:34863)
0924 04:50:41.879155 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:41.879159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:41.879414 (+ 255us) server_negotiation.cc:407] Connection header received
0924 04:50:41.879557 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:41.879561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:41.879606 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:41.879672 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:41.879966 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.880582 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.881721 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.882106 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.882196 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.882253 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:41.882492 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 04:50:41.882568 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:50:41.882720 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.878865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46629 (local address 127.0.98.1:34491)
0924 04:50:41.879223 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:41.879237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:41.879352 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:41.879663 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:41.879666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:41.879682 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:41.879820 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.879826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.880668 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.880672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.881598 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.881607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.882219 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.882223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.882365 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.882395 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:50:41.882457 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:50:41.882556 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":118,"thread_start_us":113,"threads_started":1}
I20260924 04:50:41.882730 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.878901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34863 (local address 127.0.98.1:52671)
0924 04:50:41.879320 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:41.879334 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:41.879449 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:41.879772 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:41.879775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:41.879789 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:41.879883 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.879888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.880696 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.880699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.881588 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:41.881597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.882218 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.882221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:41.882367 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.882397 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:50:41.882459 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:50:41.882556 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":37,"thread_start_us":151,"threads_started":1}
I20260924 04:50:41.883374 19314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47124f69e6a475994cd8f6455e309ac" candidate_uuid: "cc4cec4a1e0d4215b620620adcfc0705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35df3f445154bae9d7a80a6f4a18a34" is_pre_election: true
I20260924 04:50:41.883505 19445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47124f69e6a475994cd8f6455e309ac" candidate_uuid: "cc4cec4a1e0d4215b620620adcfc0705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecf33d9c1849b59e455655bcfed662" is_pre_election: true
I20260924 04:50:41.883698 19314 raft_consensus.cc:2454] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4cec4a1e0d4215b620620adcfc0705 in term 0.
I20260924 04:50:41.883776 19445 raft_consensus.cc:2454] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4cec4a1e0d4215b620620adcfc0705 in term 0.
I20260924 04:50:41.884156 19118 leader_election.cc:302] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35df3f445154bae9d7a80a6f4a18a34, cc4cec4a1e0d4215b620620adcfc0705; no voters:
I20260924 04:50:41.884446 19516 raft_consensus.cc:2781] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:50:41.884533 19516 raft_consensus.cc:482] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:50:41.884578 19516 raft_consensus.cc:3037] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:41.885556 19516 raft_consensus.cc:504] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35df3f445154bae9d7a80a6f4a18a34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46629 } } peers { permanent_uuid: "92ecf33d9c1849b59e455655bcfed662" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34863 } } peers { permanent_uuid: "cc4cec4a1e0d4215b620620adcfc0705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46747 } }
I20260924 04:50:41.885886 19516 leader_election.cc:288] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [CANDIDATE]: Term 1 election: Requested vote from peers b35df3f445154bae9d7a80a6f4a18a34 (127.0.98.2:46629), 92ecf33d9c1849b59e455655bcfed662 (127.0.98.3:34863)
I20260924 04:50:41.886214 19314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47124f69e6a475994cd8f6455e309ac" candidate_uuid: "cc4cec4a1e0d4215b620620adcfc0705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35df3f445154bae9d7a80a6f4a18a34"
I20260924 04:50:41.886214 19445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47124f69e6a475994cd8f6455e309ac" candidate_uuid: "cc4cec4a1e0d4215b620620adcfc0705" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecf33d9c1849b59e455655bcfed662"
I20260924 04:50:41.886343 19445 raft_consensus.cc:3037] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:41.886365 19314 raft_consensus.cc:3037] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:41.887624 19445 raft_consensus.cc:2454] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4cec4a1e0d4215b620620adcfc0705 in term 1.
I20260924 04:50:41.887624 19314 raft_consensus.cc:2454] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4cec4a1e0d4215b620620adcfc0705 in term 1.
I20260924 04:50:41.887928 19117 leader_election.cc:302] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ecf33d9c1849b59e455655bcfed662, cc4cec4a1e0d4215b620620adcfc0705; no voters:
I20260924 04:50:41.888074 19516 raft_consensus.cc:2781] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:41.888269 19516 raft_consensus.cc:686] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [term 1 LEADER]: Becoming Leader. State: Replica: cc4cec4a1e0d4215b620620adcfc0705, State: Running, Role: LEADER
I20260924 04:50:41.888587 19516 consensus_queue.cc:237] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35df3f445154bae9d7a80a6f4a18a34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46629 } } peers { permanent_uuid: "92ecf33d9c1849b59e455655bcfed662" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34863 } } peers { permanent_uuid: "cc4cec4a1e0d4215b620620adcfc0705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46747 } }
I20260924 04:50:41.891815 19040 catalog_manager.cc:5495] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4cec4a1e0d4215b620620adcfc0705 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cc4cec4a1e0d4215b620620adcfc0705" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35df3f445154bae9d7a80a6f4a18a34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92ecf33d9c1849b59e455655bcfed662" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc4cec4a1e0d4215b620620adcfc0705" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46747 } health_report { overall_health: HEALTHY } } }
I20260924 04:50:41.925812 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.922359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38438 (local address 127.0.98.1:46747)
0924 04:50:41.922488 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:41.922492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:41.922506 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:50:41.922579 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:41.922581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:41.922654 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:41.922748 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:50:41.923014 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.923574 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.924472 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:41.924690 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:41.924780 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:41.925023 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:50:41.925449 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:50:41.925561 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:41.925620 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 04:50:41.925657 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:50:41.931911 19445 raft_consensus.cc:1261] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662 [term 1 FOLLOWER]: Refusing update from remote peer cc4cec4a1e0d4215b620620adcfc0705: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:50:41.931911 19314 raft_consensus.cc:1261] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34 [term 1 FOLLOWER]: Refusing update from remote peer cc4cec4a1e0d4215b620620adcfc0705: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:50:41.932652 19516 consensus_queue.cc:1035] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35df3f445154bae9d7a80a6f4a18a34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:50:41.932883 19519 consensus_queue.cc:1035] T b47124f69e6a475994cd8f6455e309ac P cc4cec4a1e0d4215b620620adcfc0705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecf33d9c1849b59e455655bcfed662" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:50:41.934335 19492 tablet.cc:2369] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:50:41.941488 19524 mvcc.cc:205] Tried to move back new op lower bound from 7332763410145357824 to 7332763409978568704. Current Snapshot: MvccSnapshot[applied={T|T < 7332763410145357824}]
I20260924 04:50:41.941480 19523 mvcc.cc:205] Tried to move back new op lower bound from 7332763410145357824 to 7332763409978568704. Current Snapshot: MvccSnapshot[applied={T|T < 7332763410145357824}]
I20260924 04:50:41.950467 19040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:50:41.950636 19040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:50:41.953112 19040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1bcbff63d724b8ab2cba4807f949811: Sending DeleteTablet for 3 replicas of tablet b47124f69e6a475994cd8f6455e309ac
I20260924 04:50:41.954110 19425 tablet_service.cc:1514] Processing DeleteTablet for tablet b47124f69e6a475994cd8f6455e309ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:41 UTC) from {username='slave'} at 127.0.0.1:49022
I20260924 04:50:41.954116 19294 tablet_service.cc:1514] Processing DeleteTablet for tablet b47124f69e6a475994cd8f6455e309ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:41 UTC) from {username='slave'} at 127.0.0.1:53300
I20260924 04:50:41.954631 19532 tablet_replica.cc:330] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662: stopping tablet replica
I20260924 04:50:41.954802 19163 tablet_service.cc:1514] Processing DeleteTablet for tablet b47124f69e6a475994cd8f6455e309ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:41 UTC) from {username='slave'} at 127.0.0.1:38422
I20260924 04:50:41.955046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 19101
I20260924 04:50:41.955159 19532 raft_consensus.cc:2229] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:50:41.955511 19532 raft_consensus.cc:2258] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:50:41.956456 19533 tablet_replica.cc:330] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34: stopping tablet replica
I20260924 04:50:41.956749 19533 raft_consensus.cc:2229] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:50:41.957053 19533 raft_consensus.cc:2258] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:50:41.957479 19532 ts_tablet_manager.cc:1905] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:50:41.958446 19533 ts_tablet_manager.cc:1905] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:50:41.960239 19532 ts_tablet_manager.cc:1918] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:50:41.960325 19532 log.cc:1196] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/b47124f69e6a475994cd8f6455e309ac
I20260924 04:50:41.960592 19532 ts_tablet_manager.cc:1939] T b47124f69e6a475994cd8f6455e309ac P 92ecf33d9c1849b59e455655bcfed662: Deleting consensus metadata
I20260924 04:50:41.960917 19533 ts_tablet_manager.cc:1918] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:50:41.961007 19533 log.cc:1196] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/b47124f69e6a475994cd8f6455e309ac
I20260924 04:50:41.961263 19533 ts_tablet_manager.cc:1939] T b47124f69e6a475994cd8f6455e309ac P b35df3f445154bae9d7a80a6f4a18a34: Deleting consensus metadata
W20260924 04:50:41.961927 19026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46747 (error 108)
I20260924 04:50:41.962105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 19232
I20260924 04:50:41.962407 19026 catalog_manager.cc:4841] TS 92ecf33d9c1849b59e455655bcfed662 (127.0.98.3:34863): tablet b47124f69e6a475994cd8f6455e309ac (table test-table [id=64475697b00442d28033e92e38a82b0a]) successfully deleted
I20260924 04:50:41.963950 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.963697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46747 (local address 127.0.0.1:38448)
0924 04:50:41.963819 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:41.963879 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 04:50:41.964087 19026 catalog_manager.cc:4584] TS cc4cec4a1e0d4215b620620adcfc0705 (127.0.98.1:46747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b47124f69e6a475994cd8f6455e309ac: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46747: connect: Connection refused (error 111)
I20260924 04:50:41.966845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 19363
I20260924 04:50:41.967847 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:41.967507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46629 (local address 127.0.0.1:53302)
0924 04:50:41.967660 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:41.967706 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 04:50:41.971648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 19010
2026-09-24T04:50:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1172 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:50:42 dist-test-slave-hvcl krb5kdc[19537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:50:42 dist-test-slave-hvcl krb5kdc[19537](info): set up 2 sockets
Sep 24 04:50:42 dist-test-slave-hvcl krb5kdc[19537](info): commencing operation
krb5kdc: starting...
W20260924 04:50:42.926841 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:50:43 dist-test-slave-hvcl krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:50:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:50:43.046746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45855
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:43.154809 19550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:43.155128 19550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:43.155190 19550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:43.159291 19550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:43.159360 19550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:43.159386 19550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:43.159406 19550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:43.159427 19550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:43.164726 19550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:43.165791 19550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:43.166823 19550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:43.175125 19558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:43.175119 19556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:43.175119 19555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:43.175359 19550 server_base.cc:1034] running on GCE node
I20260924 04:50:43.176074 19550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:43.177270 19550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:43.178472 19550 hybrid_clock.cc:648] HybridClock initialized: now 1790225443178443 us; error 37 us; skew 500 ppm
Sep 24 04:50:43 dist-test-slave-hvcl krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:43.181936 19550 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:50:43.183068 19550 webserver.cc:460] Webserver started at http://127.0.98.62:35655/ using document root <none> and password file <none>
I20260924 04:50:43.183786 19550 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:43.183881 19550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:43.184130 19550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:43.186451 19550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "fb69b42d8bc5417d985ffa53045a6cd4"
format_stamp: "Formatted at 2026-09-24 04:50:43 on dist-test-slave-hvcl"
server_key: "ed0c9ba847fcf476c3a29fb10237afcb"
server_key_iv: "5f2c54066134207109b8b3bec03aa818"
server_key_version: "encryptionkey@0"
I20260924 04:50:43.187031 19550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb69b42d8bc5417d985ffa53045a6cd4"
format_stamp: "Formatted at 2026-09-24 04:50:43 on dist-test-slave-hvcl"
server_key: "ed0c9ba847fcf476c3a29fb10237afcb"
server_key_iv: "5f2c54066134207109b8b3bec03aa818"
server_key_version: "encryptionkey@0"
I20260924 04:50:43.190752 19550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:50:43.193176 19565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:43.194448 19550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:50:43.194562 19550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "fb69b42d8bc5417d985ffa53045a6cd4"
format_stamp: "Formatted at 2026-09-24 04:50:43 on dist-test-slave-hvcl"
server_key: "ed0c9ba847fcf476c3a29fb10237afcb"
server_key_iv: "5f2c54066134207109b8b3bec03aa818"
server_key_version: "encryptionkey@0"
I20260924 04:50:43.194669 19550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:43.234120 19550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:43.237352 19550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:43.237540 19550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:43.244511 19550 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45855
I20260924 04:50:43.244546 19617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45855 every 8 connection(s)
I20260924 04:50:43.245451 19550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:43.248332 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:43.253505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 19550
I20260924 04:50:43.253651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:43.253961 392 external_mini_cluster.cc:1428] Setting key c726b1826dd6de5ce988b59b281d85e1
I20260924 04:50:43.254122 19618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4: Bootstrap starting.
I20260924 04:50:43.256639 19618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:43.257373 19618 log.cc:824] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:43.259693 19618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4: No bootstrap required, opened a new log
Sep 24 04:50:43 dist-test-slave-hvcl krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:50:43.263481 19618 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb69b42d8bc5417d985ffa53045a6cd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45855 } }
I20260924 04:50:43.263780 19618 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:43.263842 19618 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb69b42d8bc5417d985ffa53045a6cd4, State: Initialized, Role: FOLLOWER
I20260924 04:50:43.264370 19618 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb69b42d8bc5417d985ffa53045a6cd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45855 } }
I20260924 04:50:43.264542 19618 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:43.264614 19618 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:43.264726 19618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:43.265825 19618 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb69b42d8bc5417d985ffa53045a6cd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45855 } }
I20260924 04:50:43.266194 19618 leader_election.cc:302] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb69b42d8bc5417d985ffa53045a6cd4; no voters:
I20260924 04:50:43.266557 19618 leader_election.cc:288] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:43.266652 19623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:43.266861 19623 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [term 1 LEADER]: Becoming Leader. State: Replica: fb69b42d8bc5417d985ffa53045a6cd4, State: Running, Role: LEADER
I20260924 04:50:43.267213 19623 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb69b42d8bc5417d985ffa53045a6cd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45855 } }
I20260924 04:50:43.267923 19618 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:43.268435 19624 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb69b42d8bc5417d985ffa53045a6cd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb69b42d8bc5417d985ffa53045a6cd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45855 } } }
I20260924 04:50:43.268563 19624 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:43.268419 19625 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb69b42d8bc5417d985ffa53045a6cd4. Latest consensus state: current_term: 1 leader_uuid: "fb69b42d8bc5417d985ffa53045a6cd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb69b42d8bc5417d985ffa53045a6cd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45855 } } }
I20260924 04:50:43.268637 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:43.255363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.0.98.62:45855)
0924 04:50:43.255781 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:43.255795 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:43.255829 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:50:43.256462 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:43.256480 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:43.256880 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:43.257200 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:43.258008 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.259445 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.260303 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.260968 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.261130 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:43.263338 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:43.263344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:43.263361 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:43.263393 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:43.265376 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:43.265776 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:43.265782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:43.265788 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:43.265933 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:43.266180 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:43.266183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:43.266185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:43.266745 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:43.266941 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:43.267319 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 04:50:43.267614 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":118,"threads_started":1}
I20260924 04:50:43.268654 19625 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:43.268939 19628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:43.272164 19628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:43.277472 19628 catalog_manager.cc:1347] Generated new cluster ID: 70309e33476248d584988a31dc380ef2
I20260924 04:50:43.277537 19628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:43.287210 19628 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:43.288266 19628 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:43.293618 19628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4: Generated new TSK 0
I20260924 04:50:43.294428 19628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:50:43.322628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45855
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:43.432608 19644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:43.432909 19644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:43.432972 19644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:43.437130 19644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:43.437207 19644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:43.437290 19644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:50:43.442757 19644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:43.443897 19644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:43.444938 19644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:43.454310 19650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:43.454310 19652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:43.454450 19649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:43.454524 19644 server_base.cc:1034] running on GCE node
I20260924 04:50:43.455137 19644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:43.455814 19644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:43.456976 19644 hybrid_clock.cc:648] HybridClock initialized: now 1790225443456953 us; error 34 us; skew 500 ppm
Sep 24 04:50:43 dist-test-slave-hvcl krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:43.459872 19644 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:50:43.460775 19644 webserver.cc:460] Webserver started at http://127.0.98.1:43669/ using document root <none> and password file <none>
I20260924 04:50:43.461315 19644 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:43.461380 19644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:43.461567 19644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:43.463658 19644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "217a04b16c284641899fae45b34fd660"
format_stamp: "Formatted at 2026-09-24 04:50:43 on dist-test-slave-hvcl"
server_key: "ad2d90ca7d5eaff6ca77d64fff0539d9"
server_key_iv: "cdbf66d7338cc16fcb50a1896777f130"
server_key_version: "encryptionkey@0"
I20260924 04:50:43.464208 19644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "217a04b16c284641899fae45b34fd660"
format_stamp: "Formatted at 2026-09-24 04:50:43 on dist-test-slave-hvcl"
server_key: "ad2d90ca7d5eaff6ca77d64fff0539d9"
server_key_iv: "cdbf66d7338cc16fcb50a1896777f130"
server_key_version: "encryptionkey@0"
I20260924 04:50:43.467963 19644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:50:43.470487 19659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:43.471832 19644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:50:43.471931 19644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "217a04b16c284641899fae45b34fd660"
format_stamp: "Formatted at 2026-09-24 04:50:43 on dist-test-slave-hvcl"
server_key: "ad2d90ca7d5eaff6ca77d64fff0539d9"
server_key_iv: "cdbf66d7338cc16fcb50a1896777f130"
server_key_version: "encryptionkey@0"
I20260924 04:50:43.472019 19644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:43.496232 19644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:43.499552 19644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:43.499730 19644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:43.500281 19644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:43.501083 19644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:43.501128 19644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:43.501166 19644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:43.501180 19644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:43.509645 19644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37647
I20260924 04:50:43.509723 19772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37647 every 8 connection(s)
I20260924 04:50:43.510552 19644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:50:43 dist-test-slave-hvcl krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:50:43.519346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 19644
I20260924 04:50:43.519479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:50:43.519744 392 external_mini_cluster.cc:1428] Setting key 8707bae0577485dce05dfc65d52f13f3
I20260924 04:50:43.523167 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:43.512320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43579 (local address 127.0.98.62:45855)
0924 04:50:43.512441 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:43.512445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:43.513364 (+ 919us) server_negotiation.cc:407] Connection header received
0924 04:50:43.514207 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:43.514210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:43.514256 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:43.514329 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:43.515503 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.516026 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.517128 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.517365 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.517440 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:43.519810 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:43.519814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:43.519817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:43.519846 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:43.521200 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:43.521725 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:43.521729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:43.521731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:43.521812 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:43.522117 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:43.522121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:43.522123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:43.522298 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:43.522387 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:43.522865 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 04:50:43.523008 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:50:43.523993 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:43.512731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45855 (local address 127.0.98.1:43579)
0924 04:50:43.513185 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:43.513229 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:43.514013 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:43.514479 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:43.514486 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:43.514829 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:43.515364 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:43.515379 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.516131 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.516134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:43.517030 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:43.517039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.517490 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.517492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:43.517634 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:43.518177 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:50:43.518193 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:43.519651 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:43.521303 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:43.521309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:43.521326 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:43.521603 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:43.521893 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:43.521896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:43.521898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:43.522036 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:43.522390 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:43.522397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:43.522647 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 04:50:43.522813 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 04:50:43.523178 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":109,"threads_started":1}
I20260924 04:50:43.525087 19773 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45855
I20260924 04:50:43.525309 19773 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:43.525758 19773 heartbeater.cc:511] Master 127.0.98.62:45855 requested a full tablet report, sending...
I20260924 04:50:43.527199 19582 ts_manager.cc:194] Registered new tserver with Master: 217a04b16c284641899fae45b34fd660 (127.0.98.1:37647)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:50:43.528303 19582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:50:43.548400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45855
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:43.655879 19778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:43.656148 19778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:43.656203 19778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:43.660292 19778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:43.660354 19778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:43.660422 19778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:50:43.665697 19778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:43.666764 19778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:43.667778 19778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:43.677469 19786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:43.677484 19783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:43.677493 19784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:43.677625 19778 server_base.cc:1034] running on GCE node
I20260924 04:50:43.678128 19778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:43.678846 19778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:43.680025 19778 hybrid_clock.cc:648] HybridClock initialized: now 1790225443680001 us; error 40 us; skew 500 ppm
Sep 24 04:50:43 dist-test-slave-hvcl krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:43.682771 19778 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:50:43.683746 19778 webserver.cc:460] Webserver started at http://127.0.98.2:36111/ using document root <none> and password file <none>
I20260924 04:50:43.684302 19778 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:43.684367 19778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:43.684553 19778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:43.686686 19778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "6b608e698ce641d9b1ab72f4c0cd0523"
format_stamp: "Formatted at 2026-09-24 04:50:43 on dist-test-slave-hvcl"
server_key: "18f4c8ac4731f6d3ed90120226053725"
server_key_iv: "edb56761332f48a4a4c3111b0653b9be"
server_key_version: "encryptionkey@0"
I20260924 04:50:43.687274 19778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6b608e698ce641d9b1ab72f4c0cd0523"
format_stamp: "Formatted at 2026-09-24 04:50:43 on dist-test-slave-hvcl"
server_key: "18f4c8ac4731f6d3ed90120226053725"
server_key_iv: "edb56761332f48a4a4c3111b0653b9be"
server_key_version: "encryptionkey@0"
I20260924 04:50:43.691146 19778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260924 04:50:43.693686 19793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:43.695029 19778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:50:43.695147 19778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "6b608e698ce641d9b1ab72f4c0cd0523"
format_stamp: "Formatted at 2026-09-24 04:50:43 on dist-test-slave-hvcl"
server_key: "18f4c8ac4731f6d3ed90120226053725"
server_key_iv: "edb56761332f48a4a4c3111b0653b9be"
server_key_version: "encryptionkey@0"
I20260924 04:50:43.695255 19778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:43.729077 19778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:43.732070 19778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:43.732247 19778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:43.732834 19778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:43.733647 19778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:43.733695 19778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:43.733747 19778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:43.733772 19778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:43.742353 19778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32943
I20260924 04:50:43.742378 19906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32943 every 8 connection(s)
I20260924 04:50:43.743291 19778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:50:43.744252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 19778
I20260924 04:50:43.744378 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:50:43.744642 392 external_mini_cluster.cc:1428] Setting key 32dee2866d1bdcf9c7ba38280c2f1d0f
Sep 24 04:50:43 dist-test-slave-hvcl krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:50:43.757468 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:43.745074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42409 (local address 127.0.98.62:45855)
0924 04:50:43.745200 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:43.745204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:43.746586 (+ 1382us) server_negotiation.cc:407] Connection header received
0924 04:50:43.747720 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:43.747723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:43.747780 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:43.747897 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:43.749051 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.749831 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.750997 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.751337 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.751448 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:43.754037 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:43.754040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:43.754042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:43.754066 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:43.755355 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:43.755920 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:43.755922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:43.755923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:43.755993 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:43.756315 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:43.756318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:43.756319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:43.756543 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:43.756634 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:43.757240 (+ 606us) server_negotiation.cc:299] Negotiation successful
0924 04:50:43.757345 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:50:43.758554 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:43.745546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45855 (local address 127.0.98.2:42409)
0924 04:50:43.746425 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:43.746483 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:43.747479 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:43.748069 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:43.748075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:43.748441 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:43.748917 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:43.748930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.749949 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.749952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:43.750867 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:43.750876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.751481 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.751484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:43.751639 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:43.752240 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:50:43.752257 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:43.753882 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:43.755481 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:43.755486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:43.755499 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:43.755843 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:43.756062 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:43.756064 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:43.756066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:43.756250 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:43.756641 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:43.756648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:43.756980 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 04:50:43.757212 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 04:50:43.757572 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":149,"threads_started":1}
I20260924 04:50:43.759757 19907 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45855
I20260924 04:50:43.760003 19907 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:43.760543 19907 heartbeater.cc:511] Master 127.0.98.62:45855 requested a full tablet report, sending...
I20260924 04:50:43.761664 19582 ts_manager.cc:194] Registered new tserver with Master: 6b608e698ce641d9b1ab72f4c0cd0523 (127.0.98.2:32943)
I20260924 04:50:43.762229 19582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:50:43.774236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45855
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:43.881645 19912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:43.881915 19912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:43.881978 19912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:43.886152 19912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:43.886228 19912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:43.886314 19912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:50:43.891806 19912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:43.892874 19912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:43.893883 19912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:43.903090 19920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:43.903187 19917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:43.903177 19918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:43.903301 19912 server_base.cc:1034] running on GCE node
I20260924 04:50:43.903815 19912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:43.904475 19912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:43.905647 19912 hybrid_clock.cc:648] HybridClock initialized: now 1790225443905629 us; error 34 us; skew 500 ppm
Sep 24 04:50:43 dist-test-slave-hvcl krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:43.908452 19912 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:50:43.909389 19912 webserver.cc:460] Webserver started at http://127.0.98.3:37233/ using document root <none> and password file <none>
I20260924 04:50:43.909950 19912 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:43.910009 19912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:43.910188 19912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:43.912293 19912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "33195298b0e44ad797ef0bcd3b923939"
format_stamp: "Formatted at 2026-09-24 04:50:43 on dist-test-slave-hvcl"
server_key: "93bb916993fbd94eeef7c37a5c405064"
server_key_iv: "2066c45bd6cf7c71b956d6943e348c5e"
server_key_version: "encryptionkey@0"
I20260924 04:50:43.912806 19912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "33195298b0e44ad797ef0bcd3b923939"
format_stamp: "Formatted at 2026-09-24 04:50:43 on dist-test-slave-hvcl"
server_key: "93bb916993fbd94eeef7c37a5c405064"
server_key_iv: "2066c45bd6cf7c71b956d6943e348c5e"
server_key_version: "encryptionkey@0"
I20260924 04:50:43.916611 19912 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:50:43.919181 19927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:43.920498 19912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:50:43.920604 19912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "33195298b0e44ad797ef0bcd3b923939"
format_stamp: "Formatted at 2026-09-24 04:50:43 on dist-test-slave-hvcl"
server_key: "93bb916993fbd94eeef7c37a5c405064"
server_key_iv: "2066c45bd6cf7c71b956d6943e348c5e"
server_key_version: "encryptionkey@0"
I20260924 04:50:43.920699 19912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:43.941475 19912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:43.944530 19912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:43.944700 19912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:43.945308 19912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:43.946141 19912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:43.946188 19912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:43.946229 19912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:43.946250 19912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:43.954676 19912 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35939
I20260924 04:50:43.954691 20040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35939 every 8 connection(s)
I20260924 04:50:43.955637 19912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:50:43.960450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 19912
I20260924 04:50:43.960568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:50:43.960834 392 external_mini_cluster.cc:1428] Setting key b991bb43b9d1f364c4dde950766a7a4e
Sep 24 04:50:43 dist-test-slave-hvcl krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:50:43.968488 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:43.957549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54467 (local address 127.0.98.62:45855)
0924 04:50:43.957680 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:43.957684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:43.958658 (+ 974us) server_negotiation.cc:407] Connection header received
0924 04:50:43.959537 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:43.959540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:43.959588 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:43.959661 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:43.960864 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.961425 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.962535 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.962754 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.962838 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:43.965175 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:43.965179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:43.965182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:43.965209 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:43.966460 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:43.966947 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:43.966951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:43.966953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:43.967072 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:43.967389 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:43.967393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:43.967395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:43.967586 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:43.967669 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:43.968186 (+ 517us) server_negotiation.cc:299] Negotiation successful
0924 04:50:43.968312 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:50:43.969357 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:43.958017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45855 (local address 127.0.98.3:54467)
0924 04:50:43.958488 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:43.958533 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:43.959341 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:43.959823 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:43.959830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:43.960186 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:43.960714 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:43.960730 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.961544 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.961547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:43.962420 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:43.962429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.962891 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.962895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:43.963074 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:43.963576 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:50:43.963592 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:43.965010 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:43.966572 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:43.966578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:43.966595 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:43.966855 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:43.967151 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:50:43.967155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:50:43.967157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:50:43.967302 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:50:43.967701 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:43.967708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:43.967963 (+ 255us) client_negotiation.cc:769] Sending connection context
0924 04:50:43.968131 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 04:50:43.968470 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":127,"threads_started":1}
I20260924 04:50:43.970484 20041 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45855
I20260924 04:50:43.970705 20041 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:43.971225 20041 heartbeater.cc:511] Master 127.0.98.62:45855 requested a full tablet report, sending...
I20260924 04:50:43.972256 19582 ts_manager.cc:194] Registered new tserver with Master: 33195298b0e44ad797ef0bcd3b923939 (127.0.98.3:35939)
I20260924 04:50:43.972836 19582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54467
I20260924 04:50:43.974537 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:50:43.983693 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:43.976571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.0.98.62:45855)
0924 04:50:43.976704 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:43.976708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:43.976861 (+ 153us) server_negotiation.cc:407] Connection header received
0924 04:50:43.976961 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:43.976963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:43.977005 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:43.977103 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:43.977712 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.978210 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.979231 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:43.979460 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:43.979550 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:43.980926 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:43.980930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:43.980933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:43.980960 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:43.982054 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:43.982450 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:43.982454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:43.982456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:43.982537 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:43.982785 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:43.982788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:43.982790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:43.983075 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:43.983203 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:43.983417 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 04:50:43.983534 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:50:43.986581 19582 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:50:43.989544 19582 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:50:44.004458 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:43.999025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37647 (local address 127.0.0.1:58748)
0924 04:50:43.999652 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:43.999664 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:43.999735 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:44.000570 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:44.000573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:44.000590 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:44.000659 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:44.000664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.002218 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.002222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:44.003090 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:44.003100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.003882 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.003886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:44.004007 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.004044 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 04:50:44.004151 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 04:50:44.004282 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":73,"thread_start_us":88,"threads_started":1}
I20260924 04:50:44.004504 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:43.998353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32943 (local address 127.0.0.1:46358)
0924 04:50:43.999261 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:43.999314 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:43.999441 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:44.000113 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:44.000119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:44.000141 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:44.000279 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:44.000286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.001877 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.001882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:44.003112 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:44.003119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.004181 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.004185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:44.004269 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.004290 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:50:44.004340 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 04:50:44.004401 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":18,"thread_start_us":93,"threads_started":1}
I20260924 04:50:44.004601 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:43.999323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.0.98.2:32943)
0924 04:50:43.999670 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:43.999683 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:43.999704 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:50:43.999774 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:43.999779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:43.999945 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:44.000427 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:44.000586 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.001749 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.003239 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.004063 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.004176 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.004315 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:44.004381 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 04:50:44.004462 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":138,"threads_started":1}
I20260924 04:50:44.004951 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:43.998262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35939 (local address 127.0.0.1:34994)
0924 04:50:43.999262 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:43.999313 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:43.999442 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:44.000258 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:44.000260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:44.000275 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:44.000355 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:44.000359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.001492 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.001495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:44.003089 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:44.003100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.004571 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.004575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:44.004660 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.004683 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:50:44.004748 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:50:44.004815 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":68,"thread_start_us":80,"threads_started":1}
I20260924 04:50:44.005155 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:43.999918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.0.98.1:37647)
0924 04:50:44.000176 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:44.000189 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:44.000208 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:50:44.000273 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:44.000277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:44.000446 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:44.000587 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:44.000751 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.002072 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.003205 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.003770 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.003875 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.004042 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:44.004907 (+ 865us) server_negotiation.cc:299] Negotiation successful
0924 04:50:44.005012 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260924 04:50:44.005606 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:43.998440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.0.98.3:35939)
0924 04:50:43.998902 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:43.998915 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:43.999587 (+ 672us) server_negotiation.cc:407] Connection header received
0924 04:50:43.999674 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:43.999680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:43.999899 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:44.000072 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:44.000464 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.001338 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.003233 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.004063 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.004175 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.004315 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:44.005350 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0924 04:50:44.005461 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":169,"threads_started":1}
I20260924 04:50:44.007763 19707 tablet_service.cc:1467] Processing CreateTablet for tablet 8222b0b7367b44d69d6a403e3ca79ce5 (DEFAULT_TABLE table=test-table [id=8e18564f09dc4ae19bbd492895bd698e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:44.008047 19975 tablet_service.cc:1467] Processing CreateTablet for tablet 8222b0b7367b44d69d6a403e3ca79ce5 (DEFAULT_TABLE table=test-table [id=8e18564f09dc4ae19bbd492895bd698e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:44.008843 19707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8222b0b7367b44d69d6a403e3ca79ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:44.008986 19975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8222b0b7367b44d69d6a403e3ca79ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:44.010360 19841 tablet_service.cc:1467] Processing CreateTablet for tablet 8222b0b7367b44d69d6a403e3ca79ce5 (DEFAULT_TABLE table=test-table [id=8e18564f09dc4ae19bbd492895bd698e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:44.011430 19841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8222b0b7367b44d69d6a403e3ca79ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:44.014824 20058 tablet_bootstrap.cc:492] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939: Bootstrap starting.
I20260924 04:50:44.016561 20059 tablet_bootstrap.cc:492] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660: Bootstrap starting.
I20260924 04:50:44.017555 20058 tablet_bootstrap.cc:654] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:44.018438 20060 tablet_bootstrap.cc:492] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523: Bootstrap starting.
I20260924 04:50:44.018630 20058 log.cc:824] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:44.018713 20059 tablet_bootstrap.cc:654] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:44.019833 20059 log.cc:824] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:44.020471 20060 tablet_bootstrap.cc:654] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:44.020707 20058 tablet_bootstrap.cc:492] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939: No bootstrap required, opened a new log
I20260924 04:50:44.020949 20058 ts_tablet_manager.cc:1397] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:50:44.021543 20060 log.cc:824] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:44.021821 20059 tablet_bootstrap.cc:492] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660: No bootstrap required, opened a new log
I20260924 04:50:44.022002 20059 ts_tablet_manager.cc:1397] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 04:50:44.023234 20060 tablet_bootstrap.cc:492] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523: No bootstrap required, opened a new log
I20260924 04:50:44.023398 20060 ts_tablet_manager.cc:1397] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:50:44.024847 20058 raft_consensus.cc:348] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33195298b0e44ad797ef0bcd3b923939" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "6b608e698ce641d9b1ab72f4c0cd0523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32943 } } peers { permanent_uuid: "217a04b16c284641899fae45b34fd660" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 04:50:44.025139 20058 raft_consensus.cc:374] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:44.025204 20058 raft_consensus.cc:729] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33195298b0e44ad797ef0bcd3b923939, State: Initialized, Role: FOLLOWER
I20260924 04:50:44.025671 20058 consensus_queue.cc:260] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33195298b0e44ad797ef0bcd3b923939" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "6b608e698ce641d9b1ab72f4c0cd0523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32943 } } peers { permanent_uuid: "217a04b16c284641899fae45b34fd660" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 04:50:44.025838 20059 raft_consensus.cc:348] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33195298b0e44ad797ef0bcd3b923939" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "6b608e698ce641d9b1ab72f4c0cd0523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32943 } } peers { permanent_uuid: "217a04b16c284641899fae45b34fd660" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 04:50:44.026088 20059 raft_consensus.cc:374] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:44.026144 20059 raft_consensus.cc:729] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217a04b16c284641899fae45b34fd660, State: Initialized, Role: FOLLOWER
I20260924 04:50:44.026474 20041 heartbeater.cc:503] Master 127.0.98.62:45855 was elected leader, sending a full tablet report...
I20260924 04:50:44.026587 20059 consensus_queue.cc:260] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33195298b0e44ad797ef0bcd3b923939" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "6b608e698ce641d9b1ab72f4c0cd0523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32943 } } peers { permanent_uuid: "217a04b16c284641899fae45b34fd660" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 04:50:44.026508 20060 raft_consensus.cc:348] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33195298b0e44ad797ef0bcd3b923939" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "6b608e698ce641d9b1ab72f4c0cd0523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32943 } } peers { permanent_uuid: "217a04b16c284641899fae45b34fd660" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 04:50:44.026721 20058 ts_tablet_manager.cc:1428] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:50:44.026744 20060 raft_consensus.cc:374] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:44.026793 20060 raft_consensus.cc:729] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b608e698ce641d9b1ab72f4c0cd0523, State: Initialized, Role: FOLLOWER
I20260924 04:50:44.027222 20060 consensus_queue.cc:260] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33195298b0e44ad797ef0bcd3b923939" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "6b608e698ce641d9b1ab72f4c0cd0523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32943 } } peers { permanent_uuid: "217a04b16c284641899fae45b34fd660" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 04:50:44.027523 20059 ts_tablet_manager.cc:1428] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260924 04:50:44.027542 19773 heartbeater.cc:503] Master 127.0.98.62:45855 was elected leader, sending a full tablet report...
I20260924 04:50:44.028169 19907 heartbeater.cc:503] Master 127.0.98.62:45855 was elected leader, sending a full tablet report...
I20260924 04:50:44.028577 20060 ts_tablet_manager.cc:1428] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:50:44.093634 20065 raft_consensus.cc:482] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:50:44.093844 20065 raft_consensus.cc:504] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33195298b0e44ad797ef0bcd3b923939" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "6b608e698ce641d9b1ab72f4c0cd0523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32943 } } peers { permanent_uuid: "217a04b16c284641899fae45b34fd660" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 04:50:44.095100 20065 leader_election.cc:288] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33195298b0e44ad797ef0bcd3b923939 (127.0.98.3:35939), 6b608e698ce641d9b1ab72f4c0cd0523 (127.0.98.2:32943)
I20260924 04:50:44.099534 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:44.095391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35939 (local address 127.0.98.1:52583)
0924 04:50:44.096022 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:44.096036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:44.096135 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:44.096441 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:44.096444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:44.096463 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:44.096578 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:44.096585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.097376 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.097379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:44.098328 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:44.098338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.099109 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.099112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:44.099220 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.099242 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:50:44.099302 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:50:44.099365 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":327,"thread_start_us":94,"threads_started":1}
I20260924 04:50:44.099543 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:44.095504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52583 (local address 127.0.98.3:35939)
0924 04:50:44.095634 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:44.095638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:44.096119 (+ 481us) server_negotiation.cc:407] Connection header received
0924 04:50:44.096266 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:44.096269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:44.096318 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:44.096382 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:44.096741 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.097248 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.098459 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.098998 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.099094 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.099161 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:44.099340 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 04:50:44.099421 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:50:44.099617 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:44.095474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59839 (local address 127.0.98.2:32943)
0924 04:50:44.095606 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:44.095610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:44.095883 (+ 273us) server_negotiation.cc:407] Connection header received
0924 04:50:44.096035 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:44.096038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:44.096078 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:44.096125 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:44.096570 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.097138 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.098400 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.099011 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.099129 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.099194 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:44.099438 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 04:50:44.099515 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:50:44.100028 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:44.095388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32943 (local address 127.0.98.1:59839)
0924 04:50:44.095765 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:44.095781 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:44.095901 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:44.096273 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:44.096276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:44.096293 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:44.096424 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:44.096431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.097252 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.097256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:44.098275 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:44.098285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.099120 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.099124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:44.099269 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.099289 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:50:44.099818 (+ 529us) client_negotiation.cc:240] Negotiation successful
0924 04:50:44.099891 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":73,"thread_start_us":140,"threads_started":1}
I20260924 04:50:44.100361 19995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8222b0b7367b44d69d6a403e3ca79ce5" candidate_uuid: "217a04b16c284641899fae45b34fd660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33195298b0e44ad797ef0bcd3b923939" is_pre_election: true
I20260924 04:50:44.100701 19995 raft_consensus.cc:2454] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 217a04b16c284641899fae45b34fd660 in term 0.
I20260924 04:50:44.100703 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8222b0b7367b44d69d6a403e3ca79ce5" candidate_uuid: "217a04b16c284641899fae45b34fd660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b608e698ce641d9b1ab72f4c0cd0523" is_pre_election: true
I20260924 04:50:44.100945 19861 raft_consensus.cc:2454] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 217a04b16c284641899fae45b34fd660 in term 0.
I20260924 04:50:44.101151 19662 leader_election.cc:302] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 217a04b16c284641899fae45b34fd660, 33195298b0e44ad797ef0bcd3b923939; no voters:
I20260924 04:50:44.101450 20065 raft_consensus.cc:2781] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:50:44.101526 20065 raft_consensus.cc:482] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:50:44.101557 20065 raft_consensus.cc:3037] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:44.102550 20065 raft_consensus.cc:504] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33195298b0e44ad797ef0bcd3b923939" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "6b608e698ce641d9b1ab72f4c0cd0523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32943 } } peers { permanent_uuid: "217a04b16c284641899fae45b34fd660" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 04:50:44.102890 20065 leader_election.cc:288] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [CANDIDATE]: Term 1 election: Requested vote from peers 33195298b0e44ad797ef0bcd3b923939 (127.0.98.3:35939), 6b608e698ce641d9b1ab72f4c0cd0523 (127.0.98.2:32943)
I20260924 04:50:44.103255 19995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8222b0b7367b44d69d6a403e3ca79ce5" candidate_uuid: "217a04b16c284641899fae45b34fd660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33195298b0e44ad797ef0bcd3b923939"
I20260924 04:50:44.103287 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8222b0b7367b44d69d6a403e3ca79ce5" candidate_uuid: "217a04b16c284641899fae45b34fd660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b608e698ce641d9b1ab72f4c0cd0523"
I20260924 04:50:44.103364 19995 raft_consensus.cc:3037] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:44.103382 19861 raft_consensus.cc:3037] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:44.104375 19995 raft_consensus.cc:2454] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 217a04b16c284641899fae45b34fd660 in term 1.
I20260924 04:50:44.104378 19861 raft_consensus.cc:2454] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 217a04b16c284641899fae45b34fd660 in term 1.
I20260924 04:50:44.104656 19660 leader_election.cc:302] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 217a04b16c284641899fae45b34fd660, 6b608e698ce641d9b1ab72f4c0cd0523; no voters:
I20260924 04:50:44.104820 20065 raft_consensus.cc:2781] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:44.105008 20065 raft_consensus.cc:686] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [term 1 LEADER]: Becoming Leader. State: Replica: 217a04b16c284641899fae45b34fd660, State: Running, Role: LEADER
I20260924 04:50:44.105335 20065 consensus_queue.cc:237] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33195298b0e44ad797ef0bcd3b923939" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } } peers { permanent_uuid: "6b608e698ce641d9b1ab72f4c0cd0523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32943 } } peers { permanent_uuid: "217a04b16c284641899fae45b34fd660" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 04:50:44.108646 19582 catalog_manager.cc:5495] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 217a04b16c284641899fae45b34fd660 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "217a04b16c284641899fae45b34fd660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33195298b0e44ad797ef0bcd3b923939" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b608e698ce641d9b1ab72f4c0cd0523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "217a04b16c284641899fae45b34fd660" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } health_report { overall_health: HEALTHY } } }
I20260924 04:50:44.128352 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:44.124986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58762 (local address 127.0.98.1:37647)
0924 04:50:44.125120 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:44.125125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:44.125139 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:50:44.125256 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:44.125260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:44.125316 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:44.125412 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:50:44.125641 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.126139 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.126992 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.127209 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.127292 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.127572 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:50:44.127988 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:50:44.128092 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:44.128153 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 04:50:44.128208 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:50:44.134162 19995 raft_consensus.cc:1261] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939 [term 1 FOLLOWER]: Refusing update from remote peer 217a04b16c284641899fae45b34fd660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:50:44.134162 19861 raft_consensus.cc:1261] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523 [term 1 FOLLOWER]: Refusing update from remote peer 217a04b16c284641899fae45b34fd660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:50:44.134861 20065 consensus_queue.cc:1035] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33195298b0e44ad797ef0bcd3b923939" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:50:44.135073 20069 consensus_queue.cc:1035] T 8222b0b7367b44d69d6a403e3ca79ce5 P 217a04b16c284641899fae45b34fd660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b608e698ce641d9b1ab72f4c0cd0523" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:50:44.142774 20074 mvcc.cc:205] Tried to move back new op lower bound from 7332763419166494720 to 7332763419058106368. Current Snapshot: MvccSnapshot[applied={T|T < 7332763419166494720}]
I20260924 04:50:44.145290 20073 mvcc.cc:205] Tried to move back new op lower bound from 7332763419166494720 to 7332763419058106368. Current Snapshot: MvccSnapshot[applied={T|T < 7332763419166494720}]
I20260924 04:50:44.150156 19582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:50:44.150309 19582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:50:44.152678 19582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb69b42d8bc5417d985ffa53045a6cd4: Sending DeleteTablet for 3 replicas of tablet 8222b0b7367b44d69d6a403e3ca79ce5
I20260924 04:50:44.153434 19975 tablet_service.cc:1514] Processing DeleteTablet for tablet 8222b0b7367b44d69d6a403e3ca79ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34994
I20260924 04:50:44.153439 19841 tablet_service.cc:1514] Processing DeleteTablet for tablet 8222b0b7367b44d69d6a403e3ca79ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46358
I20260924 04:50:44.153975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 19644
I20260924 04:50:44.154042 19707 tablet_service.cc:1514] Processing DeleteTablet for tablet 8222b0b7367b44d69d6a403e3ca79ce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58748
I20260924 04:50:44.154683 20083 tablet_replica.cc:330] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523: stopping tablet replica
I20260924 04:50:44.155023 20083 raft_consensus.cc:2229] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:50:44.155324 20083 raft_consensus.cc:2258] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:50:44.155666 20082 tablet_replica.cc:330] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939: stopping tablet replica
I20260924 04:50:44.155937 20082 raft_consensus.cc:2229] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:50:44.156195 20082 raft_consensus.cc:2258] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:50:44.157367 20082 ts_tablet_manager.cc:1905] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:50:44.157389 20083 ts_tablet_manager.cc:1905] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:50:44.159485 20082 ts_tablet_manager.cc:1918] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:50:44.159569 20082 log.cc:1196] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/8222b0b7367b44d69d6a403e3ca79ce5
I20260924 04:50:44.159818 20082 ts_tablet_manager.cc:1939] T 8222b0b7367b44d69d6a403e3ca79ce5 P 33195298b0e44ad797ef0bcd3b923939: Deleting consensus metadata
I20260924 04:50:44.160009 20083 ts_tablet_manager.cc:1918] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:50:44.160101 20083 log.cc:1196] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/8222b0b7367b44d69d6a403e3ca79ce5
I20260924 04:50:44.160385 20083 ts_tablet_manager.cc:1939] T 8222b0b7367b44d69d6a403e3ca79ce5 P 6b608e698ce641d9b1ab72f4c0cd0523: Deleting consensus metadata
I20260924 04:50:44.160977 19568 catalog_manager.cc:4841] TS 33195298b0e44ad797ef0bcd3b923939 (127.0.98.3:35939): tablet 8222b0b7367b44d69d6a403e3ca79ce5 (table test-table [id=8e18564f09dc4ae19bbd492895bd698e]) successfully deleted
W20260924 04:50:44.161551 19568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37647 (error 108)
I20260924 04:50:44.161594 19566 catalog_manager.cc:4841] TS 6b608e698ce641d9b1ab72f4c0cd0523 (127.0.98.2:32943): tablet 8222b0b7367b44d69d6a403e3ca79ce5 (table test-table [id=8e18564f09dc4ae19bbd492895bd698e]) successfully deleted
I20260924 04:50:44.161805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 19778
I20260924 04:50:44.162719 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:44.162449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37647 (local address 127.0.0.1:58774)
0924 04:50:44.162575 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:44.162639 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 04:50:44.162886 19568 catalog_manager.cc:4584] TS 217a04b16c284641899fae45b34fd660 (127.0.98.1:37647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8222b0b7367b44d69d6a403e3ca79ce5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37647: connect: Connection refused (error 111)
I20260924 04:50:44.167371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 19912
I20260924 04:50:44.172137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 19550
2026-09-24T04:50:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2201 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T04:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:44Z Disabled control of system clock
I20260924 04:50:44.200172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43879
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:50:44.307400 20088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:44.307694 20088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:44.307729 20088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:44.311787 20088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:44.311856 20088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:44.311872 20088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:44.311887 20088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:44.311900 20088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:44.316744 20088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:44.317863 20088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:44.319289 20088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:44.327628 20094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:44.327661 20093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:44.327699 20088 server_base.cc:1034] running on GCE node
W20260924 04:50:44.327834 20096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:44.328403 20088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:44.329481 20088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:44.330631 20088 hybrid_clock.cc:648] HybridClock initialized: now 1790225444330612 us; error 30 us; skew 500 ppm
I20260924 04:50:44.332444 20088 webserver.cc:460] Webserver started at http://127.0.98.62:37391/ using document root <none> and password file <none>
I20260924 04:50:44.332979 20088 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:44.333039 20088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:44.333228 20088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:44.335244 20088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "970f7d9f2c094c9cb36578913b30a70c"
format_stamp: "Formatted at 2026-09-24 04:50:44 on dist-test-slave-hvcl"
server_key: "c19cef015dbb8fbdd99f5095cdd29939"
server_key_iv: "1cfcb3c2f6bf4d3c6a17ca8460c2ad25"
server_key_version: "encryptionkey@0"
I20260924 04:50:44.335776 20088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "970f7d9f2c094c9cb36578913b30a70c"
format_stamp: "Formatted at 2026-09-24 04:50:44 on dist-test-slave-hvcl"
server_key: "c19cef015dbb8fbdd99f5095cdd29939"
server_key_iv: "1cfcb3c2f6bf4d3c6a17ca8460c2ad25"
server_key_version: "encryptionkey@0"
I20260924 04:50:44.339354 20088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 04:50:44.341753 20102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:44.343029 20088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:50:44.343138 20088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "970f7d9f2c094c9cb36578913b30a70c"
format_stamp: "Formatted at 2026-09-24 04:50:44 on dist-test-slave-hvcl"
server_key: "c19cef015dbb8fbdd99f5095cdd29939"
server_key_iv: "1cfcb3c2f6bf4d3c6a17ca8460c2ad25"
server_key_version: "encryptionkey@0"
I20260924 04:50:44.343237 20088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:44.359700 20088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:44.360440 20088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:44.360621 20088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:44.367483 20088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43879
I20260924 04:50:44.367508 20154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43879 every 8 connection(s)
I20260924 04:50:44.368420 20088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:44.371652 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:44.375779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 20088
I20260924 04:50:44.375900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:44.376143 392 external_mini_cluster.cc:1428] Setting key ebb6c52b7791a597f3b57abfe7f8b313
I20260924 04:50:44.377450 20155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c: Bootstrap starting.
I20260924 04:50:44.379766 20155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:44.380522 20155 log.cc:824] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:44.382359 20155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c: No bootstrap required, opened a new log
I20260924 04:50:44.384943 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:44.377546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.0.98.62:43879)
0924 04:50:44.378152 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:44.378167 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:44.378204 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:50:44.378830 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:44.378849 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:44.379295 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:44.379611 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:44.380004 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.381011 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.381857 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.382323 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.382476 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.383276 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:44.383281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:44.383294 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:44.383676 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:44.383731 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:44.383752 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:44.383832 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:44.383964 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 04:50:44.384225 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260924 04:50:44.386040 20155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970f7d9f2c094c9cb36578913b30a70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43879 } }
I20260924 04:50:44.386317 20155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:44.386375 20155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 970f7d9f2c094c9cb36578913b30a70c, State: Initialized, Role: FOLLOWER
I20260924 04:50:44.386838 20155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970f7d9f2c094c9cb36578913b30a70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43879 } }
I20260924 04:50:44.387012 20155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:44.387079 20155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:44.387151 20155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:44.388015 20155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970f7d9f2c094c9cb36578913b30a70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43879 } }
I20260924 04:50:44.388351 20155 leader_election.cc:302] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 970f7d9f2c094c9cb36578913b30a70c; no voters:
I20260924 04:50:44.388638 20155 leader_election.cc:288] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:44.388796 20160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:44.389022 20160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [term 1 LEADER]: Becoming Leader. State: Replica: 970f7d9f2c094c9cb36578913b30a70c, State: Running, Role: LEADER
I20260924 04:50:44.389333 20160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970f7d9f2c094c9cb36578913b30a70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43879 } }
I20260924 04:50:44.389878 20155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:44.390743 20162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 970f7d9f2c094c9cb36578913b30a70c. Latest consensus state: current_term: 1 leader_uuid: "970f7d9f2c094c9cb36578913b30a70c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970f7d9f2c094c9cb36578913b30a70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43879 } } }
I20260924 04:50:44.391170 20162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:44.391032 20161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "970f7d9f2c094c9cb36578913b30a70c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970f7d9f2c094c9cb36578913b30a70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43879 } } }
I20260924 04:50:44.391516 20161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:44.391583 20169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:44.394702 20169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:44.400267 20169 catalog_manager.cc:1347] Generated new cluster ID: 7785a1704a514e768a1f05e34b2587a8
I20260924 04:50:44.400352 20169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:44.438835 20169 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:44.440182 20169 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:44.448797 20169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c: Generated new TSK 0
I20260924 04:50:44.449599 20169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:50:44.463287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43879
--builtin_ntp_servers=127.0.98.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:50:44.594832 20179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:44.595146 20179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:44.595212 20179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:44.599162 20179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:44.599222 20179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:44.599277 20179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:50:44.603984 20179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:44.605031 20179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:44.606362 20179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:44.615976 20187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:44.615969 20184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:44.616025 20179 server_base.cc:1034] running on GCE node
W20260924 04:50:44.615974 20185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:44.616562 20179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:44.617223 20179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:44.618398 20179 hybrid_clock.cc:648] HybridClock initialized: now 1790225444618365 us; error 42 us; skew 500 ppm
I20260924 04:50:44.620308 20179 webserver.cc:460] Webserver started at http://127.0.98.1:34963/ using document root <none> and password file <none>
I20260924 04:50:44.620862 20179 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:44.620927 20179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:44.621116 20179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:44.623282 20179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/data/instance:
uuid: "e36fdc8ee77c40fb8386f6ca04e064f9"
format_stamp: "Formatted at 2026-09-24 04:50:44 on dist-test-slave-hvcl"
server_key: "0d6360827581edddcc0a6b5fbfac716c"
server_key_iv: "8d7632679e1765b38f27818bca2813a2"
server_key_version: "encryptionkey@0"
I20260924 04:50:44.623837 20179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e36fdc8ee77c40fb8386f6ca04e064f9"
format_stamp: "Formatted at 2026-09-24 04:50:44 on dist-test-slave-hvcl"
server_key: "0d6360827581edddcc0a6b5fbfac716c"
server_key_iv: "8d7632679e1765b38f27818bca2813a2"
server_key_version: "encryptionkey@0"
I20260924 04:50:44.627543 20179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:50:44.629961 20193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:44.631291 20179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:50:44.631404 20179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/wal
uuid: "e36fdc8ee77c40fb8386f6ca04e064f9"
format_stamp: "Formatted at 2026-09-24 04:50:44 on dist-test-slave-hvcl"
server_key: "0d6360827581edddcc0a6b5fbfac716c"
server_key_iv: "8d7632679e1765b38f27818bca2813a2"
server_key_version: "encryptionkey@0"
I20260924 04:50:44.631510 20179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:44.676796 20179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:44.677496 20179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:44.677671 20179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:44.678267 20179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:44.679203 20179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:44.679250 20179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:44.679299 20179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:44.679322 20179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:44.687667 20179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35805
I20260924 04:50:44.687691 20306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35805 every 8 connection(s)
I20260924 04:50:44.688577 20179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:50:44.690261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 20179
I20260924 04:50:44.690414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:50:44.690758 392 external_mini_cluster.cc:1428] Setting key 27494aa85fabc7f7e620417595865b46
I20260924 04:50:44.692899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43879
--builtin_ntp_servers=127.0.98.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:50:44.698581 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:44.690511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50131 (local address 127.0.98.62:43879)
0924 04:50:44.690669 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:44.690674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:44.691982 (+ 1308us) server_negotiation.cc:407] Connection header received
0924 04:50:44.693799 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:44.693803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:44.693853 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:44.693934 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:44.694961 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.695520 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.696738 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.696961 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.697049 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.698116 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:44.698120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:44.698123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:44.698137 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:44.698155 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:44.698163 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:44.698223 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:44.698361 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 04:50:44.698426 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:50:44.699532 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:44.691029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43879 (local address 127.0.98.1:50131)
0924 04:50:44.691849 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:44.691892 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:44.693553 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:44.694119 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:44.694129 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:44.694258 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:44.694821 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:44.694834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.695645 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.695649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:44.696620 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:44.696630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.697103 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.697107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:44.697275 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.697859 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:50:44.697861 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:44.697903 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:50:44.697970 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:44.698232 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:44.698241 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:44.698247 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:50:44.698355 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 04:50:44.698640 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":105,"threads_started":1}
I20260924 04:50:44.700886 20307 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43879
I20260924 04:50:44.701116 20307 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:44.701634 20307 heartbeater.cc:511] Master 127.0.98.62:43879 requested a full tablet report, sending...
I20260924 04:50:44.703222 20119 ts_manager.cc:194] Registered new tserver with Master: e36fdc8ee77c40fb8386f6ca04e064f9 (127.0.98.1:35805)
I20260924 04:50:44.704763 20119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50131
W20260924 04:50:44.803290 20310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:44.803571 20310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:44.803610 20310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:44.807646 20310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:44.807704 20310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:44.807760 20310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:50:44.812634 20310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:44.813681 20310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:44.815030 20310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:44.824914 20316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:44.825016 20310 server_base.cc:1034] running on GCE node
W20260924 04:50:44.824946 20318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:44.824949 20315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:44.825554 20310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:44.826172 20310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:44.827322 20310 hybrid_clock.cc:648] HybridClock initialized: now 1790225444827301 us; error 33 us; skew 500 ppm
I20260924 04:50:44.829067 20310 webserver.cc:460] Webserver started at http://127.0.98.2:37061/ using document root <none> and password file <none>
I20260924 04:50:44.829612 20310 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:44.829675 20310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:44.829859 20310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:44.831908 20310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/data/instance:
uuid: "59bd00aa6f4146b09d9534d4c1de2abb"
format_stamp: "Formatted at 2026-09-24 04:50:44 on dist-test-slave-hvcl"
server_key: "8ef5e0d48b1335630963ce369ff62af0"
server_key_iv: "cdb9486cdf817ec62b7b2f5d9caa2d21"
server_key_version: "encryptionkey@0"
I20260924 04:50:44.832424 20310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/wal/instance:
uuid: "59bd00aa6f4146b09d9534d4c1de2abb"
format_stamp: "Formatted at 2026-09-24 04:50:44 on dist-test-slave-hvcl"
server_key: "8ef5e0d48b1335630963ce369ff62af0"
server_key_iv: "cdb9486cdf817ec62b7b2f5d9caa2d21"
server_key_version: "encryptionkey@0"
I20260924 04:50:44.836091 20310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:50:44.838471 20324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:44.839725 20310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:50:44.839840 20310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/wal
uuid: "59bd00aa6f4146b09d9534d4c1de2abb"
format_stamp: "Formatted at 2026-09-24 04:50:44 on dist-test-slave-hvcl"
server_key: "8ef5e0d48b1335630963ce369ff62af0"
server_key_iv: "cdb9486cdf817ec62b7b2f5d9caa2d21"
server_key_version: "encryptionkey@0"
I20260924 04:50:44.839941 20310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:44.853463 20310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:44.854198 20310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:44.854379 20310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:44.855000 20310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:44.855978 20310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:44.856024 20310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:44.856068 20310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:44.856097 20310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:44.864705 20310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40429
I20260924 04:50:44.864727 20437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40429 every 8 connection(s)
I20260924 04:50:44.865626 20310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:50:44.869146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 20310
I20260924 04:50:44.869268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:50:44.869555 392 external_mini_cluster.cc:1428] Setting key a4dfcafea1391f492349e41cb5dc00da
I20260924 04:50:44.871497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43879
--builtin_ntp_servers=127.0.98.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:50:44.875777 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:44.867431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35777 (local address 127.0.98.62:43879)
0924 04:50:44.867584 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:44.867588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:44.868703 (+ 1115us) server_negotiation.cc:407] Connection header received
0924 04:50:44.870285 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:44.870289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:44.870345 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:44.870417 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:44.871499 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.872079 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.873864 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.874087 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.874196 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.875314 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:44.875317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:44.875319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:44.875332 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:44.875351 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:44.875359 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:44.875417 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:44.875560 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 04:50:44.875622 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:50:44.876703 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:44.867829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43879 (local address 127.0.98.2:35777)
0924 04:50:44.868523 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:44.868577 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:44.870070 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:44.870623 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:44.870631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:44.870745 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:44.871363 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:44.871377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.872203 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.872206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:44.873654 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:44.873666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:44.874241 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:44.874246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:44.874433 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:44.875044 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:50:44.875046 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:44.875092 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:50:44.875164 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:44.875428 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:44.875438 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:44.875443 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:50:44.875554 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 04:50:44.875895 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":104,"threads_started":1}
I20260924 04:50:44.877960 20438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43879
I20260924 04:50:44.878217 20438 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:44.878862 20438 heartbeater.cc:511] Master 127.0.98.62:43879 requested a full tablet report, sending...
I20260924 04:50:44.880012 20119 ts_manager.cc:194] Registered new tserver with Master: 59bd00aa6f4146b09d9534d4c1de2abb (127.0.98.2:40429)
I20260924 04:50:44.880796 20119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35777
W20260924 04:50:44.981861 20441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:44.982153 20441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:44.982192 20441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:44.986222 20441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:44.986292 20441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:44.986348 20441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:50:44.991258 20441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:44.992383 20441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:44.993729 20441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:45.003273 20446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:45.003274 20447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:45.003372 20441 server_base.cc:1034] running on GCE node
W20260924 04:50:45.003286 20449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:45.003970 20441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:45.004616 20441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:45.005821 20441 hybrid_clock.cc:648] HybridClock initialized: now 1790225445005804 us; error 33 us; skew 500 ppm
I20260924 04:50:45.007864 20441 webserver.cc:460] Webserver started at http://127.0.98.3:42653/ using document root <none> and password file <none>
I20260924 04:50:45.008456 20441 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:45.008522 20441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:45.008738 20441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:45.010861 20441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/data/instance:
uuid: "91ca61f249ad48959069b53c5c4ffbdd"
format_stamp: "Formatted at 2026-09-24 04:50:45 on dist-test-slave-hvcl"
server_key: "00727cca84014d46d4ab6089e6ba1201"
server_key_iv: "6465ccbaab47ee9a91bde52d02fa21d2"
server_key_version: "encryptionkey@0"
I20260924 04:50:45.011447 20441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/wal/instance:
uuid: "91ca61f249ad48959069b53c5c4ffbdd"
format_stamp: "Formatted at 2026-09-24 04:50:45 on dist-test-slave-hvcl"
server_key: "00727cca84014d46d4ab6089e6ba1201"
server_key_iv: "6465ccbaab47ee9a91bde52d02fa21d2"
server_key_version: "encryptionkey@0"
I20260924 04:50:45.015231 20441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:50:45.017796 20455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:45.019126 20441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:50:45.019243 20441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/wal
uuid: "91ca61f249ad48959069b53c5c4ffbdd"
format_stamp: "Formatted at 2026-09-24 04:50:45 on dist-test-slave-hvcl"
server_key: "00727cca84014d46d4ab6089e6ba1201"
server_key_iv: "6465ccbaab47ee9a91bde52d02fa21d2"
server_key_version: "encryptionkey@0"
I20260924 04:50:45.019351 20441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:45.034000 20441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:45.034705 20441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:45.034878 20441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:45.035497 20441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:50:45.036635 20441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:50:45.036695 20441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:45.036756 20441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:50:45.036784 20441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:45.047724 20441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38839
I20260924 04:50:45.047740 20568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38839 every 8 connection(s)
I20260924 04:50:45.048676 20441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:50:45.050280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 20441
I20260924 04:50:45.050393 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:50:45.050796 392 external_mini_cluster.cc:1428] Setting key 2a5856e0ae2b676cfe814aa3cc90382b
I20260924 04:50:45.057706 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.050444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58399 (local address 127.0.98.62:43879)
0924 04:50:45.050572 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:45.050575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:45.051947 (+ 1372us) server_negotiation.cc:407] Connection header received
0924 04:50:45.053272 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:45.053274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:45.053318 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:45.053377 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:45.054295 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.054801 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.055950 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.056178 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.056253 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:45.057256 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:45.057259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:45.057261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:45.057271 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:45.057285 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:45.057291 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:45.057341 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:45.057540 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 04:50:45.057591 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:50:45.058596 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.051049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43879 (local address 127.0.98.3:58399)
0924 04:50:45.051795 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:45.051847 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:45.053088 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:45.053528 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:45.053535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:45.053621 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:50:45.054185 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:45.054197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.054917 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.054920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:45.055861 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:45.055870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.056307 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.056310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:45.056458 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:45.057032 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:50:45.057033 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:50:45.057075 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:50:45.057141 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:50:45.057396 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:50:45.057405 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:50:45.057410 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:50:45.057528 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 04:50:45.057788 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":105,"threads_started":1}
I20260924 04:50:45.059859 20569 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43879
I20260924 04:50:45.060086 20569 heartbeater.cc:464] Registering TS with master...
I20260924 04:50:45.060595 20569 heartbeater.cc:511] Master 127.0.98.62:43879 requested a full tablet report, sending...
I20260924 04:50:45.061591 20119 ts_manager.cc:194] Registered new tserver with Master: 91ca61f249ad48959069b53c5c4ffbdd (127.0.98.3:38839)
I20260924 04:50:45.062140 20119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58399
I20260924 04:50:45.064558 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:50:45.070147 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.066524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.0.98.62:43879)
0924 04:50:45.066646 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:45.066650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:45.066821 (+ 171us) server_negotiation.cc:407] Connection header received
0924 04:50:45.066936 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:45.066939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:45.067016 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:45.067081 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:45.067368 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.067874 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.068703 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.068919 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.069008 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:45.069761 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:45.069764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:45.069766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:45.069778 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:45.069796 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:45.069804 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:45.069863 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:45.069950 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 04:50:45.070009 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:50:45.072911 20119 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:50:45.075874 20119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:50:45.090018 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.084474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38839 (local address 127.0.0.1:50510)
0924 04:50:45.085149 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:45.085204 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:45.085334 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:45.086101 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:45.086106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:45.086139 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:45.086279 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:45.086286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.087329 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.087331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:45.088368 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:45.088376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.089463 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.089466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:45.089569 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:45.089609 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 04:50:45.089719 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 04:50:45.089856 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":2,"spinlock_wait_cycles":12032,"thread_start_us":72,"threads_started":1}
I20260924 04:50:45.090026 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.084977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.0.98.3:38839)
0924 04:50:45.085628 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:45.085641 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:45.085663 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:50:45.085793 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:45.085797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:45.085953 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:45.086094 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:45.086409 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.087216 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.088478 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.089320 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.089464 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:45.089660 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:45.089760 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 04:50:45.089862 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":139,"threads_started":1}
I20260924 04:50:45.090408 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.084259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40429 (local address 127.0.0.1:34116)
0924 04:50:45.085149 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:45.085204 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:45.085334 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:45.086101 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:45.086111 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:45.086144 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:45.086304 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:45.086311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.087854 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.087856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:45.089081 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:45.089089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.090043 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.090046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:45.090138 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:45.090165 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:50:45.090213 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 04:50:45.090272 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":35,"spinlock_wait_cycles":18176,"thread_start_us":99,"threads_started":1}
I20260924 04:50:45.090914 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.084187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35805 (local address 127.0.0.1:37706)
0924 04:50:45.085301 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:45.085314 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:45.085392 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:45.086101 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:45.086111 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:45.086144 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:45.086376 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:45.086382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.087852 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.087856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:45.089127 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:45.089136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.090305 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.090309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:45.090404 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:45.090425 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:50:45.090683 (+ 258us) client_negotiation.cc:240] Negotiation successful
0924 04:50:45.090786 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":264,"spinlock_wait_cycles":12416,"thread_start_us":96,"threads_started":1}
I20260924 04:50:45.091071 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.084814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.0.98.2:40429)
0924 04:50:45.085620 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:45.085632 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:45.085656 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:50:45.085787 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:45.085791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:45.085953 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:45.086437 (+ 484us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:45.086574 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.087742 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.089208 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.089918 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.090576 (+ 658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:45.090771 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:45.090823 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 04:50:45.090906 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":100,"threads_started":1}
I20260924 04:50:45.091332 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.084324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37706 (local address 127.0.98.1:35805)
0924 04:50:45.085620 (+ 1296us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:45.085632 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:45.085656 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:50:45.085735 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:45.085741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:45.085953 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:45.086214 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:45.086485 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.087721 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.089260 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.090097 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.090267 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:45.090496 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:45.091076 (+ 580us) server_negotiation.cc:299] Negotiation successful
0924 04:50:45.091174 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1193,"thread_start_us":116,"threads_started":1}
I20260924 04:50:45.093611 20503 tablet_service.cc:1467] Processing CreateTablet for tablet 4427e9844fa84c0c9ccae501eb01f8f7 (DEFAULT_TABLE table=test-table [id=9a21aba7790f4753bd4fce3c74d2960c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:45.093762 20371 tablet_service.cc:1467] Processing CreateTablet for tablet 4427e9844fa84c0c9ccae501eb01f8f7 (DEFAULT_TABLE table=test-table [id=9a21aba7790f4753bd4fce3c74d2960c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:45.094156 20241 tablet_service.cc:1467] Processing CreateTablet for tablet 4427e9844fa84c0c9ccae501eb01f8f7 (DEFAULT_TABLE table=test-table [id=9a21aba7790f4753bd4fce3c74d2960c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:50:45.094576 20503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4427e9844fa84c0c9ccae501eb01f8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:45.094755 20371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4427e9844fa84c0c9ccae501eb01f8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:45.095036 20241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4427e9844fa84c0c9ccae501eb01f8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:45.101485 20586 tablet_bootstrap.cc:492] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd: Bootstrap starting.
I20260924 04:50:45.102102 20588 tablet_bootstrap.cc:492] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb: Bootstrap starting.
I20260924 04:50:45.102101 20587 tablet_bootstrap.cc:492] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9: Bootstrap starting.
I20260924 04:50:45.103744 20586 tablet_bootstrap.cc:654] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:45.104205 20587 tablet_bootstrap.cc:654] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:45.104206 20588 tablet_bootstrap.cc:654] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:45.104460 20586 log.cc:824] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:45.105010 20587 log.cc:824] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:45.105010 20588 log.cc:824] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:45.106441 20586 tablet_bootstrap.cc:492] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd: No bootstrap required, opened a new log
I20260924 04:50:45.106633 20586 ts_tablet_manager.cc:1397] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:50:45.106895 20587 tablet_bootstrap.cc:492] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9: No bootstrap required, opened a new log
I20260924 04:50:45.106917 20588 tablet_bootstrap.cc:492] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb: No bootstrap required, opened a new log
I20260924 04:50:45.107100 20587 ts_tablet_manager.cc:1397] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 04:50:45.107100 20588 ts_tablet_manager.cc:1397] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:50:45.109885 20586 raft_consensus.cc:348] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36fdc8ee77c40fb8386f6ca04e064f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35805 } } peers { permanent_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40429 } } peers { permanent_uuid: "91ca61f249ad48959069b53c5c4ffbdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260924 04:50:45.110119 20586 raft_consensus.cc:374] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:45.110189 20586 raft_consensus.cc:729] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91ca61f249ad48959069b53c5c4ffbdd, State: Initialized, Role: FOLLOWER
I20260924 04:50:45.110648 20586 consensus_queue.cc:260] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36fdc8ee77c40fb8386f6ca04e064f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35805 } } peers { permanent_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40429 } } peers { permanent_uuid: "91ca61f249ad48959069b53c5c4ffbdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260924 04:50:45.111352 20569 heartbeater.cc:503] Master 127.0.98.62:43879 was elected leader, sending a full tablet report...
I20260924 04:50:45.111265 20587 raft_consensus.cc:348] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36fdc8ee77c40fb8386f6ca04e064f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35805 } } peers { permanent_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40429 } } peers { permanent_uuid: "91ca61f249ad48959069b53c5c4ffbdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260924 04:50:45.111549 20587 raft_consensus.cc:374] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:45.111603 20587 raft_consensus.cc:729] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e36fdc8ee77c40fb8386f6ca04e064f9, State: Initialized, Role: FOLLOWER
I20260924 04:50:45.111722 20586 ts_tablet_manager.cc:1428] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260924 04:50:45.112051 20587 consensus_queue.cc:260] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36fdc8ee77c40fb8386f6ca04e064f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35805 } } peers { permanent_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40429 } } peers { permanent_uuid: "91ca61f249ad48959069b53c5c4ffbdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260924 04:50:45.112418 20588 raft_consensus.cc:348] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36fdc8ee77c40fb8386f6ca04e064f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35805 } } peers { permanent_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40429 } } peers { permanent_uuid: "91ca61f249ad48959069b53c5c4ffbdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260924 04:50:45.112661 20588 raft_consensus.cc:374] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:45.112704 20588 raft_consensus.cc:729] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59bd00aa6f4146b09d9534d4c1de2abb, State: Initialized, Role: FOLLOWER
I20260924 04:50:45.112696 20307 heartbeater.cc:503] Master 127.0.98.62:43879 was elected leader, sending a full tablet report...
I20260924 04:50:45.113032 20587 ts_tablet_manager.cc:1428] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:50:45.113389 20588 consensus_queue.cc:260] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36fdc8ee77c40fb8386f6ca04e064f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35805 } } peers { permanent_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40429 } } peers { permanent_uuid: "91ca61f249ad48959069b53c5c4ffbdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260924 04:50:45.114288 20588 ts_tablet_manager.cc:1428] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:50:45.114602 20438 heartbeater.cc:503] Master 127.0.98.62:43879 was elected leader, sending a full tablet report...
W20260924 04:50:45.116503 20439 tablet.cc:2369] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:50:45.189893 20308 tablet.cc:2369] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:50:45.299610 20570 tablet.cc:2369] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:50:45.358372 20594 raft_consensus.cc:482] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:50:45.358570 20594 raft_consensus.cc:504] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36fdc8ee77c40fb8386f6ca04e064f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35805 } } peers { permanent_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40429 } } peers { permanent_uuid: "91ca61f249ad48959069b53c5c4ffbdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260924 04:50:45.359754 20594 leader_election.cc:288] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e36fdc8ee77c40fb8386f6ca04e064f9 (127.0.98.1:35805), 91ca61f249ad48959069b53c5c4ffbdd (127.0.98.3:38839)
I20260924 04:50:45.364298 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.360143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35575 (local address 127.0.98.1:35805)
0924 04:50:45.360304 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:45.360308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:45.360871 (+ 563us) server_negotiation.cc:407] Connection header received
0924 04:50:45.360962 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:45.360964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:45.361014 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:45.361084 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:45.361281 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.361908 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.363230 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.363813 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.363907 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:45.363952 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:45.364119 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 04:50:45.364186 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 04:50:45.364298 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.360148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43703 (local address 127.0.98.3:38839)
0924 04:50:45.360296 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:45.360300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:45.360317 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:50:45.360463 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:45.360466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:45.360520 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:45.360594 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:50:45.360904 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.361643 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.363073 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.363674 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.363776 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:45.363842 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:45.364085 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 04:50:45.364164 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:50:45.364302 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.360067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38839 (local address 127.0.98.2:43703)
0924 04:50:45.360197 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:45.360212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:45.360314 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:45.360627 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:45.360631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:45.360655 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:45.360773 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:45.360781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.361748 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.361750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:45.362941 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:45.362952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.363809 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.363811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:45.363910 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:45.363981 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 04:50:45.364049 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:50:45.364124 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":33}
I20260924 04:50:45.364310 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.360080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35805 (local address 127.0.98.2:35575)
0924 04:50:45.360773 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:45.360786 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:50:45.360880 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:50:45.361097 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:50:45.361099 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:50:45.361110 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:50:45.361193 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:45.361197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.361996 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.361999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:45.363122 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:50:45.363129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.363908 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.363911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:50:45.364012 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:45.364030 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:50:45.364087 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 04:50:45.364133 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":362,"thread_start_us":117,"threads_started":1}
I20260924 04:50:45.365043 20261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4427e9844fa84c0c9ccae501eb01f8f7" candidate_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36fdc8ee77c40fb8386f6ca04e064f9" is_pre_election: true
I20260924 04:50:45.365056 20509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4427e9844fa84c0c9ccae501eb01f8f7" candidate_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ca61f249ad48959069b53c5c4ffbdd" is_pre_election: true
I20260924 04:50:45.365317 20509 raft_consensus.cc:2454] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59bd00aa6f4146b09d9534d4c1de2abb in term 0.
I20260924 04:50:45.365324 20261 raft_consensus.cc:2454] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59bd00aa6f4146b09d9534d4c1de2abb in term 0.
I20260924 04:50:45.365798 20325 leader_election.cc:302] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59bd00aa6f4146b09d9534d4c1de2abb, e36fdc8ee77c40fb8386f6ca04e064f9; no voters:
I20260924 04:50:45.366063 20594 raft_consensus.cc:2781] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:50:45.366148 20594 raft_consensus.cc:482] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:50:45.366195 20594 raft_consensus.cc:3037] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:45.367301 20594 raft_consensus.cc:504] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36fdc8ee77c40fb8386f6ca04e064f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35805 } } peers { permanent_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40429 } } peers { permanent_uuid: "91ca61f249ad48959069b53c5c4ffbdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260924 04:50:45.367686 20594 leader_election.cc:288] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [CANDIDATE]: Term 1 election: Requested vote from peers e36fdc8ee77c40fb8386f6ca04e064f9 (127.0.98.1:35805), 91ca61f249ad48959069b53c5c4ffbdd (127.0.98.3:38839)
I20260924 04:50:45.368134 20509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4427e9844fa84c0c9ccae501eb01f8f7" candidate_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ca61f249ad48959069b53c5c4ffbdd"
I20260924 04:50:45.368134 20261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4427e9844fa84c0c9ccae501eb01f8f7" candidate_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36fdc8ee77c40fb8386f6ca04e064f9"
I20260924 04:50:45.368264 20509 raft_consensus.cc:3037] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:45.368264 20261 raft_consensus.cc:3037] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:45.369514 20261 raft_consensus.cc:2454] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59bd00aa6f4146b09d9534d4c1de2abb in term 1.
I20260924 04:50:45.369514 20509 raft_consensus.cc:2454] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59bd00aa6f4146b09d9534d4c1de2abb in term 1.
I20260924 04:50:45.369831 20326 leader_election.cc:302] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59bd00aa6f4146b09d9534d4c1de2abb, 91ca61f249ad48959069b53c5c4ffbdd; no voters:
I20260924 04:50:45.370011 20594 raft_consensus.cc:2781] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:45.370224 20594 raft_consensus.cc:686] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [term 1 LEADER]: Becoming Leader. State: Replica: 59bd00aa6f4146b09d9534d4c1de2abb, State: Running, Role: LEADER
I20260924 04:50:45.370564 20594 consensus_queue.cc:237] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36fdc8ee77c40fb8386f6ca04e064f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35805 } } peers { permanent_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40429 } } peers { permanent_uuid: "91ca61f249ad48959069b53c5c4ffbdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } }
I20260924 04:50:45.374053 20117 catalog_manager.cc:5495] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb reported cstate change: term changed from 0 to 1, leader changed from <none> to 59bd00aa6f4146b09d9534d4c1de2abb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36fdc8ee77c40fb8386f6ca04e064f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59bd00aa6f4146b09d9534d4c1de2abb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ca61f249ad48959069b53c5c4ffbdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 } health_report { overall_health: UNKNOWN } } }
I20260924 04:50:45.454224 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.450734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.0.98.2:40429)
0924 04:50:45.450855 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:45.450859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:45.450871 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:50:45.450946 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:45.450948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:45.451029 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:45.451130 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:50:45.451394 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.451934 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.452849 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:45.453084 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:45.453182 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:45.453434 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:50:45.453852 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:50:45.453940 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:45.454027 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 04:50:45.454078 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:50:45.460156 20261 raft_consensus.cc:1261] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9 [term 1 FOLLOWER]: Refusing update from remote peer 59bd00aa6f4146b09d9534d4c1de2abb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:50:45.460156 20509 raft_consensus.cc:1261] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd [term 1 FOLLOWER]: Refusing update from remote peer 59bd00aa6f4146b09d9534d4c1de2abb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:50:45.460867 20596 consensus_queue.cc:1035] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ca61f249ad48959069b53c5c4ffbdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:50:45.461040 20594 consensus_queue.cc:1035] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e36fdc8ee77c40fb8386f6ca04e064f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:50:45.470170 20601 mvcc.cc:205] Tried to move back new op lower bound from 7332763424597729280 to 7332763424240857088. Current Snapshot: MvccSnapshot[applied={T|T < 7332763424597729280}]
I20260924 04:50:45.475275 20119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:50:45.475425 20119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:50:45.477734 20119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 970f7d9f2c094c9cb36578913b30a70c: Sending DeleteTablet for 3 replicas of tablet 4427e9844fa84c0c9ccae501eb01f8f7
I20260924 04:50:45.478492 20241 tablet_service.cc:1514] Processing DeleteTablet for tablet 4427e9844fa84c0c9ccae501eb01f8f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:45 UTC) from {username='slave'} at 127.0.0.1:37706
I20260924 04:50:45.478490 20371 tablet_service.cc:1514] Processing DeleteTablet for tablet 4427e9844fa84c0c9ccae501eb01f8f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:45 UTC) from {username='slave'} at 127.0.0.1:34116
I20260924 04:50:45.478579 20503 tablet_service.cc:1514] Processing DeleteTablet for tablet 4427e9844fa84c0c9ccae501eb01f8f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:50:45 UTC) from {username='slave'} at 127.0.0.1:50510
I20260924 04:50:45.479104 20609 tablet_replica.cc:330] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb: stopping tablet replica
I20260924 04:50:45.479211 20611 tablet_replica.cc:330] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd: stopping tablet replica
I20260924 04:50:45.479450 20609 raft_consensus.cc:2229] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:50:45.479465 20611 raft_consensus.cc:2229] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:50:45.479774 20610 tablet_replica.cc:330] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9: stopping tablet replica
I20260924 04:50:45.479822 20609 raft_consensus.cc:2258] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:50:45.479846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 20179
I20260924 04:50:45.480064 20610 raft_consensus.cc:2229] T 4427e9844fa84c0c9ccae501eb01f8f7 P e36fdc8ee77c40fb8386f6ca04e064f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:50:45.482363 20609 ts_tablet_manager.cc:1905] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:50:45.482908 20611 raft_consensus.cc:2258] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:50:45.484568 20611 ts_tablet_manager.cc:1905] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:50:45.485093 20609 ts_tablet_manager.cc:1918] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:50:45.485184 20609 log.cc:1196] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-1/wal/wals/4427e9844fa84c0c9ccae501eb01f8f7
I20260924 04:50:45.485469 20609 ts_tablet_manager.cc:1939] T 4427e9844fa84c0c9ccae501eb01f8f7 P 59bd00aa6f4146b09d9534d4c1de2abb: Deleting consensus metadata
I20260924 04:50:45.486730 20105 catalog_manager.cc:4841] TS 59bd00aa6f4146b09d9534d4c1de2abb (127.0.98.2:40429): tablet 4427e9844fa84c0c9ccae501eb01f8f7 (table test-table [id=9a21aba7790f4753bd4fce3c74d2960c]) successfully deleted
W20260924 04:50:45.486814 20103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35805 (error 108)
I20260924 04:50:45.487637 20611 ts_tablet_manager.cc:1918] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:50:45.487763 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:45.487504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35805 (local address 127.0.0.1:37720)
0924 04:50:45.487620 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 04:50:45.487681 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 04:50:45.487848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 20310
I20260924 04:50:45.487876 20611 log.cc:1196] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd: Deleting WAL directory at /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225194677594-392-0/minicluster-data/ts-2/wal/wals/4427e9844fa84c0c9ccae501eb01f8f7
I20260924 04:50:45.488161 20611 ts_tablet_manager.cc:1939] T 4427e9844fa84c0c9ccae501eb01f8f7 P 91ca61f249ad48959069b53c5c4ffbdd: Deleting consensus metadata
W20260924 04:50:45.489102 20103 catalog_manager.cc:4584] TS e36fdc8ee77c40fb8386f6ca04e064f9 (127.0.98.1:35805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4427e9844fa84c0c9ccae501eb01f8f7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35805: connect: Connection refused (error 111)
I20260924 04:50:45.489565 20104 catalog_manager.cc:4841] TS 91ca61f249ad48959069b53c5c4ffbdd (127.0.98.3:38839): tablet 4427e9844fa84c0c9ccae501eb01f8f7 (table test-table [id=9a21aba7790f4753bd4fce3c74d2960c]) successfully deleted
I20260924 04:50:45.493479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 20441
I20260924 04:50:45.498257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 20088
2026-09-24T04:50:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1325 ms)
[----------] 6 tests from EncryptionPolicyTest (10119 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:50:45 dist-test-slave-hvcl krb5kdc[20615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:50:45 dist-test-slave-hvcl krb5kdc[20615](info): set up 2 sockets
Sep 24 04:50:45 dist-test-slave-hvcl krb5kdc[20615](info): commencing operation
krb5kdc: starting...
W20260924 04:50:46.458901 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:50:46 dist-test-slave-hvcl krb5kdc[20615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:50:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:50:46.578598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:46.688467 20628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:46.688772 20628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:46.688834 20628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:46.693019 20628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:46.693097 20628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:46.693125 20628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:46.693146 20628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:46.693172 20628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:46.699255 20628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:46.700379 20628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:46.701429 20628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:46.709976 20633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:46.709992 20636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:46.710000 20634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:46.710327 20628 server_base.cc:1034] running on GCE node
I20260924 04:50:46.710865 20628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:46.712064 20628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:46.713218 20628 hybrid_clock.cc:648] HybridClock initialized: now 1790225446713200 us; error 33 us; skew 500 ppm
Sep 24 04:50:46 dist-test-slave-hvcl krb5kdc[20615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:46.716152 20628 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:50:46.717041 20628 webserver.cc:460] Webserver started at http://127.0.0.1:36979/ using document root <none> and password file <none>
I20260924 04:50:46.717562 20628 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:46.717619 20628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:46.717823 20628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:46.719897 20628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "af86be109640482b97c5d935fd8aca49"
format_stamp: "Formatted at 2026-09-24 04:50:46 on dist-test-slave-hvcl"
server_key: "68ab6d087daf3b9bd7c45439acb8a491"
server_key_iv: "416bbf8935b4e21e215e297c4aeb6099"
server_key_version: "encryptionkey@0"
I20260924 04:50:46.720414 20628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "af86be109640482b97c5d935fd8aca49"
format_stamp: "Formatted at 2026-09-24 04:50:46 on dist-test-slave-hvcl"
server_key: "68ab6d087daf3b9bd7c45439acb8a491"
server_key_iv: "416bbf8935b4e21e215e297c4aeb6099"
server_key_version: "encryptionkey@0"
I20260924 04:50:46.724059 20628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:50:46.726413 20643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:46.727919 20628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:50:46.728032 20628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "af86be109640482b97c5d935fd8aca49"
format_stamp: "Formatted at 2026-09-24 04:50:46 on dist-test-slave-hvcl"
server_key: "68ab6d087daf3b9bd7c45439acb8a491"
server_key_iv: "416bbf8935b4e21e215e297c4aeb6099"
server_key_version: "encryptionkey@0"
I20260924 04:50:46.728132 20628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:46.745361 20628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:46.748241 20628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:46.748425 20628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:46.755096 20628 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40351
I20260924 04:50:46.755116 20695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40351 every 8 connection(s)
I20260924 04:50:46.756002 20628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:46.758635 20696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:46.764217 20696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49: Bootstrap starting.
I20260924 04:50:46.764715 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 20628
I20260924 04:50:46.764871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:46.765110 392 external_mini_cluster.cc:1428] Setting key 42814722578511b1fdee7e1386928ebb
I20260924 04:50:46.766800 20696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:46.767521 20696 log.cc:824] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:46.769667 20696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49: No bootstrap required, opened a new log
Sep 24 04:50:46 dist-test-slave-hvcl krb5kdc[20615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:50:46.773955 20696 raft_consensus.cc:348] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af86be109640482b97c5d935fd8aca49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } }
I20260924 04:50:46.774216 20696 raft_consensus.cc:374] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:46.774276 20696 raft_consensus.cc:729] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af86be109640482b97c5d935fd8aca49, State: Initialized, Role: FOLLOWER
I20260924 04:50:46.774731 20696 consensus_queue.cc:260] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af86be109640482b97c5d935fd8aca49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } }
I20260924 04:50:46.774873 20696 raft_consensus.cc:388] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:46.774937 20696 raft_consensus.cc:482] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:46.775054 20696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:46.776188 20696 raft_consensus.cc:504] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af86be109640482b97c5d935fd8aca49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } }
I20260924 04:50:46.776576 20696 leader_election.cc:302] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af86be109640482b97c5d935fd8aca49; no voters:
I20260924 04:50:46.776913 20696 leader_election.cc:288] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:46.777040 20701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:46.777235 20701 raft_consensus.cc:686] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [term 1 LEADER]: Becoming Leader. State: Replica: af86be109640482b97c5d935fd8aca49, State: Running, Role: LEADER
I20260924 04:50:46.777554 20701 consensus_queue.cc:237] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af86be109640482b97c5d935fd8aca49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } }
I20260924 04:50:46.778293 20696 sys_catalog.cc:564] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:46.778790 20703 sys_catalog.cc:455] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af86be109640482b97c5d935fd8aca49. Latest consensus state: current_term: 1 leader_uuid: "af86be109640482b97c5d935fd8aca49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af86be109640482b97c5d935fd8aca49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } } }
I20260924 04:50:46.778916 20703 sys_catalog.cc:458] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:46.779179 20702 sys_catalog.cc:455] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af86be109640482b97c5d935fd8aca49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af86be109640482b97c5d935fd8aca49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40351 } } }
I20260924 04:50:46.779254 20702 sys_catalog.cc:458] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:46.779506 20705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:46.780288 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:46.766419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59209 (local address 127.0.0.1:40351)
0924 04:50:46.766783 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:46.766798 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:46.766834 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:50:46.767512 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:46.767528 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:46.767901 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:46.768264 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:46.769055 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:46.770084 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:46.771258 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:46.771752 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:46.771864 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:46.774619 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:46.774624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:46.774638 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:46.774666 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:46.776544 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:46.777134 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:46.777141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:46.777148 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:46.777295 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:46.777734 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:46.777736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:46.777738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:46.778245 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:46.778472 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:46.778901 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 04:50:46.779192 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":94,"threads_started":1}
I20260924 04:50:46.783157 20705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:46.784421 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:50:46.788494 20705 catalog_manager.cc:1347] Generated new cluster ID: 98de53b5b3004f19912f548edf875e89
I20260924 04:50:46.788565 20705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:46.796422 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:46.786683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58627 (local address 127.0.0.1:40351)
0924 04:50:46.786787 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:46.786791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:46.787102 (+ 311us) server_negotiation.cc:407] Connection header received
0924 04:50:46.787316 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:46.787319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:46.787378 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:46.787462 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:46.788322 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:46.789132 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:46.790356 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:46.790754 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:46.790834 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:46.792722 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:46.792726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:46.792729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:46.792760 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:46.794278 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:46.794861 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:46.794865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:46.794866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:46.794955 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:46.795389 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:46.795392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:46.795394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:46.795608 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:46.795735 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:46.796149 (+ 414us) server_negotiation.cc:299] Negotiation successful
0924 04:50:46.796255 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:50:46.797999 20705 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:46.798888 20705 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:46.807408 20705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af86be109640482b97c5d935fd8aca49: Generated new TSK 0
I20260924 04:50:46.808125 20705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:50:46.816768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 20628
W20260924 04:50:46.821444 20613 connection.cc:535] client connection to 127.0.0.1:40351 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40351): Connection reset by peer (error 104)
2026-09-24T04:50:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:50:46 dist-test-slave-hvcl krb5kdc[20727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:50:46 dist-test-slave-hvcl krb5kdc[20727](info): set up 2 sockets
Sep 24 04:50:46 dist-test-slave-hvcl krb5kdc[20727](info): commencing operation
krb5kdc: starting...
W20260924 04:50:47.774856 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:50:47 dist-test-slave-hvcl krb5kdc[20727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:50:47.899029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:48.011910 20740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:48.012214 20740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:48.012257 20740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:48.016459 20740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:48.016533 20740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:48.016552 20740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:48.016568 20740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:48.016582 20740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:48.022193 20740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37169
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:48.023368 20740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:48.024434 20740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:48.033056 20748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:48.033049 20745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:48.033015 20746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:48.033211 20740 server_base.cc:1034] running on GCE node
I20260924 04:50:48.033831 20740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:48.035015 20740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:48.036206 20740 hybrid_clock.cc:648] HybridClock initialized: now 1790225448036163 us; error 37 us; skew 500 ppm
Sep 24 04:50:48 dist-test-slave-hvcl krb5kdc[20727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:48.038877 20740 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:50:48.039804 20740 webserver.cc:460] Webserver started at http://127.0.0.1:40299/ using document root <none> and password file <none>
I20260924 04:50:48.040334 20740 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:48.040391 20740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:48.040550 20740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:48.042551 20740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "fea4203558ff44b2a4eb8656e70dddab"
format_stamp: "Formatted at 2026-09-24 04:50:48 on dist-test-slave-hvcl"
server_key: "06a911adbb8c0f89c97aedca4273295e"
server_key_iv: "33be039e67a0d5a157a5a48fee81a2f8"
server_key_version: "encryptionkey@0"
I20260924 04:50:48.043088 20740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "fea4203558ff44b2a4eb8656e70dddab"
format_stamp: "Formatted at 2026-09-24 04:50:48 on dist-test-slave-hvcl"
server_key: "06a911adbb8c0f89c97aedca4273295e"
server_key_iv: "33be039e67a0d5a157a5a48fee81a2f8"
server_key_version: "encryptionkey@0"
I20260924 04:50:48.046883 20740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:50:48.049530 20755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:48.050858 20740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:50:48.051017 20740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "fea4203558ff44b2a4eb8656e70dddab"
format_stamp: "Formatted at 2026-09-24 04:50:48 on dist-test-slave-hvcl"
server_key: "06a911adbb8c0f89c97aedca4273295e"
server_key_iv: "33be039e67a0d5a157a5a48fee81a2f8"
server_key_version: "encryptionkey@0"
I20260924 04:50:48.051132 20740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:48.062917 20740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:48.066385 20740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:48.066541 20740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:48.073411 20740 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37169
I20260924 04:50:48.073437 20807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37169 every 8 connection(s)
I20260924 04:50:48.074445 20740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:48.075443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 20740
I20260924 04:50:48.075578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:48.075855 392 external_mini_cluster.cc:1428] Setting key 2c833b8791a625a3e350c7e068590374
I20260924 04:50:48.077922 20808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:48.084208 20808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab: Bootstrap starting.
I20260924 04:50:48.085501 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:48.077346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47739 (local address 127.0.0.1:37169)
0924 04:50:48.077872 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:48.077888 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:48.078062 (+ 174us) server_negotiation.cc:407] Connection header received
0924 04:50:48.078895 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:48.078914 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:48.079339 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:48.079636 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:48.079990 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:48.080987 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:48.082142 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:48.082618 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:48.082747 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:48.083718 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:48.083723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:48.083737 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:48.084125 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:48.084192 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:48.084220 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:48.084331 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:48.084556 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 04:50:48.084698 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":151,"threads_started":1}
I20260924 04:50:48.086555 20808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:48.087420 20808 log.cc:824] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:48.088599 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:50:48.089380 20808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab: No bootstrap required, opened a new log
I20260924 04:50:48.093011 20808 raft_consensus.cc:348] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea4203558ff44b2a4eb8656e70dddab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } }
I20260924 04:50:48.093268 20808 raft_consensus.cc:374] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:48.093348 20808 raft_consensus.cc:729] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea4203558ff44b2a4eb8656e70dddab, State: Initialized, Role: FOLLOWER
I20260924 04:50:48.093791 20808 consensus_queue.cc:260] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea4203558ff44b2a4eb8656e70dddab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } }
I20260924 04:50:48.093916 20808 raft_consensus.cc:388] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:48.093976 20808 raft_consensus.cc:482] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:48.094074 20808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:48.094411 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:48.090499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55071 (local address 127.0.0.1:37169)
0924 04:50:48.090625 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:48.090630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:48.090832 (+ 202us) server_negotiation.cc:407] Connection header received
0924 04:50:48.090961 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:48.090964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:48.091024 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:48.091087 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:48.091353 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:48.091886 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:48.092704 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:48.092929 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:48.093013 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:48.093859 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:48.093864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:48.093866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:48.093880 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:48.093910 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:48.093925 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:48.094000 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:48.094233 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 04:50:48.094276 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:50:48.095192 20808 raft_consensus.cc:504] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea4203558ff44b2a4eb8656e70dddab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } }
I20260924 04:50:48.095551 20808 leader_election.cc:302] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fea4203558ff44b2a4eb8656e70dddab; no voters:
I20260924 04:50:48.095831 20808 leader_election.cc:288] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:48.095920 20818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:48.096127 20818 raft_consensus.cc:686] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [term 1 LEADER]: Becoming Leader. State: Replica: fea4203558ff44b2a4eb8656e70dddab, State: Running, Role: LEADER
I20260924 04:50:48.096557 20818 consensus_queue.cc:237] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea4203558ff44b2a4eb8656e70dddab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } }
I20260924 04:50:48.096998 20808 sys_catalog.cc:564] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:48.098215 20819 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fea4203558ff44b2a4eb8656e70dddab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea4203558ff44b2a4eb8656e70dddab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } } }
I20260924 04:50:48.098259 20820 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [sys.catalog]: SysCatalogTable state changed. Reason: New leader fea4203558ff44b2a4eb8656e70dddab. Latest consensus state: current_term: 1 leader_uuid: "fea4203558ff44b2a4eb8656e70dddab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea4203558ff44b2a4eb8656e70dddab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37169 } } }
I20260924 04:50:48.098371 20820 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:48.098318 20819 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab [sys.catalog]: This master's current role is: LEADER
W20260924 04:50:48.101487 20833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:50:48.101558 20833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:50:48.101673 20829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:48.102392 20829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:48.108141 20829 catalog_manager.cc:1347] Generated new cluster ID: dfefe370010146b58266d84b620b6d21
I20260924 04:50:48.108237 20829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:48.132165 20829 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:48.133137 20829 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:48.136861 20829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fea4203558ff44b2a4eb8656e70dddab: Generated new TSK 0
I20260924 04:50:48.137544 20829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:50:48.141399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 20740
W20260924 04:50:48.145963 20725 connection.cc:535] client connection to 127.0.0.1:37169 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37169): Connection reset by peer (error 104)
2026-09-24T04:50:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:50:48 dist-test-slave-hvcl krb5kdc[20839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:50:48 dist-test-slave-hvcl krb5kdc[20839](info): set up 2 sockets
Sep 24 04:50:48 dist-test-slave-hvcl krb5kdc[20839](info): commencing operation
krb5kdc: starting...
W20260924 04:50:49.090922 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:50:49 dist-test-slave-hvcl krb5kdc[20839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:50:49.211859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:49.320710 20852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:49.321013 20852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:49.321056 20852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:49.325217 20852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:49.325307 20852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:49.325326 20852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:49.325343 20852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:49.325358 20852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:49.331082 20852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:49.332217 20852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:49.333459 20852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:49.342226 20860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:49.342226 20858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:49.342226 20857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:49.342653 20852 server_base.cc:1034] running on GCE node
I20260924 04:50:49.343307 20852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:49.344465 20852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:49.345634 20852 hybrid_clock.cc:648] HybridClock initialized: now 1790225449345588 us; error 62 us; skew 500 ppm
Sep 24 04:50:49 dist-test-slave-hvcl krb5kdc[20839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:49.348554 20852 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:50:49.349593 20852 webserver.cc:460] Webserver started at http://127.0.0.1:37759/ using document root <none> and password file <none>
I20260924 04:50:49.350248 20852 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:49.350322 20852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:49.350526 20852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:49.352672 20852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "31004719e5e24cc2ac3716f371ac4599"
format_stamp: "Formatted at 2026-09-24 04:50:49 on dist-test-slave-hvcl"
server_key: "06852321050ab30d5fb2f1ea5169ac9d"
server_key_iv: "a39aafd8a915faed4001668512a46779"
server_key_version: "encryptionkey@0"
I20260924 04:50:49.353211 20852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "31004719e5e24cc2ac3716f371ac4599"
format_stamp: "Formatted at 2026-09-24 04:50:49 on dist-test-slave-hvcl"
server_key: "06852321050ab30d5fb2f1ea5169ac9d"
server_key_iv: "a39aafd8a915faed4001668512a46779"
server_key_version: "encryptionkey@0"
I20260924 04:50:49.356988 20852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 04:50:49.359673 20867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:49.361049 20852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:50:49.361173 20852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "31004719e5e24cc2ac3716f371ac4599"
format_stamp: "Formatted at 2026-09-24 04:50:49 on dist-test-slave-hvcl"
server_key: "06852321050ab30d5fb2f1ea5169ac9d"
server_key_iv: "a39aafd8a915faed4001668512a46779"
server_key_version: "encryptionkey@0"
I20260924 04:50:49.361297 20852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:49.365845 20852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:49.368948 20852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:49.369117 20852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:49.375797 20852 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41529
I20260924 04:50:49.375833 20919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41529 every 8 connection(s)
I20260924 04:50:49.376744 20852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:49.377640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 20852
I20260924 04:50:49.377777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:49.378052 392 external_mini_cluster.cc:1428] Setting key 2caf090b2f2099277598dbc07b4386b7
I20260924 04:50:49.379912 20920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:49.383795 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:49.379498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37673 (local address 127.0.0.1:41529)
0924 04:50:49.379924 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:49.379943 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:49.379989 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:50:49.380848 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:49.380871 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:49.381234 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:49.381606 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:49.382265 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:49.382270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:49.382286 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:49.382297 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:49.382360 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:49.382392 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:49.382543 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:49.382614 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 04:50:49.382774 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":107,"threads_started":1}
I20260924 04:50:49.386790 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:50:49.387015 20920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599: Bootstrap starting.
I20260924 04:50:49.389659 20920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:49.390393 20920 log.cc:824] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:49.390841 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:49.389146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54401 (local address 127.0.0.1:41529)
0924 04:50:49.389289 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:49.389294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:49.389481 (+ 187us) server_negotiation.cc:407] Connection header received
0924 04:50:49.389612 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:49.389615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:49.389651 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:49.389713 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:49.390439 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:49.390442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:49.390445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:49.390457 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:49.390485 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:49.390495 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:49.390575 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:49.390693 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 04:50:49.390725 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:50:49.392690 20920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599: No bootstrap required, opened a new log
I20260924 04:50:49.396446 20920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31004719e5e24cc2ac3716f371ac4599" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41529 } }
I20260924 04:50:49.396718 20920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:49.396780 20920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31004719e5e24cc2ac3716f371ac4599, State: Initialized, Role: FOLLOWER
I20260924 04:50:49.397207 20920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31004719e5e24cc2ac3716f371ac4599" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41529 } }
I20260924 04:50:49.397354 20920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:49.397421 20920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:49.397526 20920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:49.398481 20920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31004719e5e24cc2ac3716f371ac4599" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41529 } }
I20260924 04:50:49.398852 20920 leader_election.cc:302] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31004719e5e24cc2ac3716f371ac4599; no voters:
I20260924 04:50:49.399209 20920 leader_election.cc:288] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:49.399346 20930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:49.399592 20930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [term 1 LEADER]: Becoming Leader. State: Replica: 31004719e5e24cc2ac3716f371ac4599, State: Running, Role: LEADER
I20260924 04:50:49.399912 20930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31004719e5e24cc2ac3716f371ac4599" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41529 } }
I20260924 04:50:49.400527 20920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:49.401633 20932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31004719e5e24cc2ac3716f371ac4599. Latest consensus state: current_term: 1 leader_uuid: "31004719e5e24cc2ac3716f371ac4599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31004719e5e24cc2ac3716f371ac4599" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41529 } } }
I20260924 04:50:49.401757 20932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:49.402040 20931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31004719e5e24cc2ac3716f371ac4599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31004719e5e24cc2ac3716f371ac4599" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41529 } } }
I20260924 04:50:49.402129 20931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:49.402227 20939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:49.405033 20939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:49.410671 20939 catalog_manager.cc:1347] Generated new cluster ID: 33b7b9108f2644ae99be22615b07d9cc
I20260924 04:50:49.410869 20939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:49.425992 20939 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:49.426357 20939 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:49.433372 20939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31004719e5e24cc2ac3716f371ac4599: Generated new TSK 0
I20260924 04:50:49.434144 20939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:50:49.444240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 20852
2026-09-24T04:50:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:50:49 dist-test-slave-hvcl krb5kdc[20951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:50:49 dist-test-slave-hvcl krb5kdc[20951](info): set up 2 sockets
Sep 24 04:50:49 dist-test-slave-hvcl krb5kdc[20951](info): commencing operation
krb5kdc: starting...
W20260924 04:50:50.394938 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:50:50 dist-test-slave-hvcl krb5kdc[20951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:50:50.513751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38881
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:50.621901 20964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:50.622202 20964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:50.622262 20964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:50.626368 20964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:50.626447 20964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:50.626473 20964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:50.626494 20964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:50.626513 20964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:50.631959 20964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:50.633049 20964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:50.634044 20964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:50.642345 20969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:50.642407 20970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:50.642365 20972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:50.642549 20964 server_base.cc:1034] running on GCE node
I20260924 04:50:50.643301 20964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:50.644461 20964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:50.645609 20964 hybrid_clock.cc:648] HybridClock initialized: now 1790225450645586 us; error 40 us; skew 500 ppm
Sep 24 04:50:50 dist-test-slave-hvcl krb5kdc[20951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:50.654922 20964 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:50:50.655987 20964 webserver.cc:460] Webserver started at http://127.0.0.1:40171/ using document root <none> and password file <none>
I20260924 04:50:50.656597 20964 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:50.656664 20964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:50.656872 20964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:50.659066 20964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "309ca17262cf4e6fa3f34dcb7832fe4d"
format_stamp: "Formatted at 2026-09-24 04:50:50 on dist-test-slave-hvcl"
server_key: "c862bbec192095cc706bf4dd330eb405"
server_key_iv: "a5567d664ea30271e1f8d6d53627d719"
server_key_version: "encryptionkey@0"
I20260924 04:50:50.659646 20964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "309ca17262cf4e6fa3f34dcb7832fe4d"
format_stamp: "Formatted at 2026-09-24 04:50:50 on dist-test-slave-hvcl"
server_key: "c862bbec192095cc706bf4dd330eb405"
server_key_iv: "a5567d664ea30271e1f8d6d53627d719"
server_key_version: "encryptionkey@0"
I20260924 04:50:50.663311 20964 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:50:50.665704 20979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:50.667012 20964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:50:50.667128 20964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "309ca17262cf4e6fa3f34dcb7832fe4d"
format_stamp: "Formatted at 2026-09-24 04:50:50 on dist-test-slave-hvcl"
server_key: "c862bbec192095cc706bf4dd330eb405"
server_key_iv: "a5567d664ea30271e1f8d6d53627d719"
server_key_version: "encryptionkey@0"
I20260924 04:50:50.667229 20964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:50.728207 20964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:50.731570 20964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:50.731770 20964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:50.738524 20964 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38881
I20260924 04:50:50.738546 21031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38881 every 8 connection(s)
I20260924 04:50:50.739476 20964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:50.740466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 20964
I20260924 04:50:50.740590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:50.740839 392 external_mini_cluster.cc:1428] Setting key e24891c6330abfe65a41def719249e2f
I20260924 04:50:50.742504 21032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:50.748857 21032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d: Bootstrap starting.
Sep 24 04:50:50 dist-test-slave-hvcl krb5kdc[20951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:50:50.751365 21032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:50.752071 21032 log.cc:824] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:50.754312 21032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d: No bootstrap required, opened a new log
I20260924 04:50:50.755299 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:50.742183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42473 (local address 127.0.0.1:38881)
0924 04:50:50.742574 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:50.742594 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:50.742634 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:50:50.743330 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:50.743349 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:50.743704 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:50.743992 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:50.744834 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:50.745847 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:50.747017 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:50.747493 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:50.747614 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:50.750397 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:50.750402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:50.750415 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:50.750445 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:50.752074 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:50.752547 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:50.752553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:50.752558 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:50.752695 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:50.753077 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:50.753080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:50.753081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:50.753502 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:50.753637 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:50.753945 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 04:50:50.754237 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":116,"threads_started":1}
I20260924 04:50:50.757402 21032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309ca17262cf4e6fa3f34dcb7832fe4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38881 } }
I20260924 04:50:50.757620 21032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:50.757683 21032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309ca17262cf4e6fa3f34dcb7832fe4d, State: Initialized, Role: FOLLOWER
I20260924 04:50:50.757962 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:50:50.758131 21032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309ca17262cf4e6fa3f34dcb7832fe4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38881 } }
I20260924 04:50:50.758283 21032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:50.758342 21032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:50.758430 21032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:50.759632 21032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309ca17262cf4e6fa3f34dcb7832fe4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38881 } }
I20260924 04:50:50.759965 21032 leader_election.cc:302] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 309ca17262cf4e6fa3f34dcb7832fe4d; no voters:
I20260924 04:50:50.760278 21032 leader_election.cc:288] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:50.760407 21042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:50.760596 21042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [term 1 LEADER]: Becoming Leader. State: Replica: 309ca17262cf4e6fa3f34dcb7832fe4d, State: Running, Role: LEADER
I20260924 04:50:50.760913 21042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309ca17262cf4e6fa3f34dcb7832fe4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38881 } }
I20260924 04:50:50.761613 21032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:50.762008 21042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 309ca17262cf4e6fa3f34dcb7832fe4d. Latest consensus state: current_term: 1 leader_uuid: "309ca17262cf4e6fa3f34dcb7832fe4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309ca17262cf4e6fa3f34dcb7832fe4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38881 } } }
I20260924 04:50:50.762166 21042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:50.762553 21047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:50.764442 21043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "309ca17262cf4e6fa3f34dcb7832fe4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309ca17262cf4e6fa3f34dcb7832fe4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38881 } } }
I20260924 04:50:50.764581 21043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:50.765841 21047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:50.769616 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:50.760096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48165 (local address 127.0.0.1:38881)
0924 04:50:50.760205 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:50.760210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:50.760380 (+ 170us) server_negotiation.cc:407] Connection header received
0924 04:50:50.760564 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:50.760567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:50.760611 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:50.760654 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:50.761524 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:50.762303 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:50.763489 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:50.763820 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:50.763913 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:50.766450 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:50.766453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:50.766456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:50.766481 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:50.768048 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:50.768429 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:50.768432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:50.768434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:50.768508 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:50.768741 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:50.768745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:50.768746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:50.768926 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:50.769035 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:50.769252 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 04:50:50.769364 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 04:50:50.771955 21047 catalog_manager.cc:1347] Generated new cluster ID: 3ec4c231a27c42ad82f1c872a3f9c3b4
I20260924 04:50:50.772025 21047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:50.801909 21047 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:50.802937 21047 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:50.807866 21047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 309ca17262cf4e6fa3f34dcb7832fe4d: Generated new TSK 0
I20260924 04:50:50.808560 21047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:50:50.819727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 20964
2026-09-24T04:50:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:50:50 dist-test-slave-hvcl krb5kdc[21063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:50:50 dist-test-slave-hvcl krb5kdc[21063](info): set up 2 sockets
Sep 24 04:50:50 dist-test-slave-hvcl krb5kdc[21063](info): commencing operation
krb5kdc: starting...
W20260924 04:50:51.770795 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:50:51 dist-test-slave-hvcl krb5kdc[21063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:50:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:50:51.890499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33011
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:52.001677 21076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:52.002007 21076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:52.002070 21076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:52.006304 21076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:52.006389 21076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:52.006418 21076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:52.006439 21076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:52.006475 21076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:52.012284 21076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:52.013427 21076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:52.014493 21076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:52.023869 21081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:52.023869 21082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:52.023869 21084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:52.024070 21076 server_base.cc:1034] running on GCE node
I20260924 04:50:52.024804 21076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:52.026026 21076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:52.027199 21076 hybrid_clock.cc:648] HybridClock initialized: now 1790225452027173 us; error 40 us; skew 500 ppm
Sep 24 04:50:52 dist-test-slave-hvcl krb5kdc[21063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:52.030333 21076 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:50:52.031468 21076 webserver.cc:460] Webserver started at http://127.0.0.1:42957/ using document root <none> and password file <none>
I20260924 04:50:52.032065 21076 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:52.032137 21076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:52.032332 21076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:52.034485 21076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "5b0a8c9e26f4497981a9dad97956d6a0"
format_stamp: "Formatted at 2026-09-24 04:50:52 on dist-test-slave-hvcl"
server_key: "a67d45cacfa729b25383e99766bd7b11"
server_key_iv: "7dd1736f3af3c03f5f4b28244c2a2a70"
server_key_version: "encryptionkey@0"
I20260924 04:50:52.035115 21076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b0a8c9e26f4497981a9dad97956d6a0"
format_stamp: "Formatted at 2026-09-24 04:50:52 on dist-test-slave-hvcl"
server_key: "a67d45cacfa729b25383e99766bd7b11"
server_key_iv: "7dd1736f3af3c03f5f4b28244c2a2a70"
server_key_version: "encryptionkey@0"
I20260924 04:50:52.039104 21076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:50:52.041735 21091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:52.043186 21076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:50:52.043321 21076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "5b0a8c9e26f4497981a9dad97956d6a0"
format_stamp: "Formatted at 2026-09-24 04:50:52 on dist-test-slave-hvcl"
server_key: "a67d45cacfa729b25383e99766bd7b11"
server_key_iv: "7dd1736f3af3c03f5f4b28244c2a2a70"
server_key_version: "encryptionkey@0"
I20260924 04:50:52.043442 21076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:52.064386 21076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:52.067728 21076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:52.067939 21076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:52.075012 21076 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33011
I20260924 04:50:52.075047 21143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33011 every 8 connection(s)
I20260924 04:50:52.075984 21076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:52.076884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 21076
I20260924 04:50:52.077008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:52.077255 392 external_mini_cluster.cc:1428] Setting key 8c576fe0e58d039879a9c3bd4c97513b
I20260924 04:50:52.079239 21144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:52.084945 21144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0: Bootstrap starting.
I20260924 04:50:52.086318 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:52.078581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50595 (local address 127.0.0.1:33011)
0924 04:50:52.079107 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:52.079124 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:52.079180 (+ 56us) server_negotiation.cc:407] Connection header received
0924 04:50:52.079879 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:52.079898 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:52.080267 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:52.080609 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:52.081011 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:52.082001 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:52.082888 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:52.083419 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:52.083566 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:52.084514 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:52.084521 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:52.084535 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:52.084939 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:52.084994 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:52.085019 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:52.085116 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:52.085311 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 04:50:52.085519 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":155,"threads_started":1}
I20260924 04:50:52.087332 21144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:52.088030 21144 log.cc:824] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:52.089002 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:50:52.090230 21144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0: No bootstrap required, opened a new log
I20260924 04:50:52.094741 21144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0a8c9e26f4497981a9dad97956d6a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33011 } }
I20260924 04:50:52.095093 21144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:52.095168 21144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0a8c9e26f4497981a9dad97956d6a0, State: Initialized, Role: FOLLOWER
I20260924 04:50:52.095371 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:52.091249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58569 (local address 127.0.0.1:33011)
0924 04:50:52.091394 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:52.091398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:52.091545 (+ 147us) server_negotiation.cc:407] Connection header received
0924 04:50:52.091700 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:52.091702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:52.091738 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:52.091800 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:52.092061 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:52.092611 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:52.093748 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:52.094012 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:52.094096 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:52.094875 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:52.094878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:52.094881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:52.094893 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:52.094917 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:52.094928 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:52.095018 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:52.095178 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 04:50:52.095240 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 04:50:52.095661 21144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0a8c9e26f4497981a9dad97956d6a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33011 } }
I20260924 04:50:52.095798 21144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:52.095863 21144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:52.095961 21144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:52.097010 21144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0a8c9e26f4497981a9dad97956d6a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33011 } }
I20260924 04:50:52.097347 21144 leader_election.cc:302] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b0a8c9e26f4497981a9dad97956d6a0; no voters:
I20260924 04:50:52.097638 21144 leader_election.cc:288] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:52.097746 21154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:52.097966 21154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [term 1 LEADER]: Becoming Leader. State: Replica: 5b0a8c9e26f4497981a9dad97956d6a0, State: Running, Role: LEADER
I20260924 04:50:52.098352 21154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0a8c9e26f4497981a9dad97956d6a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33011 } }
I20260924 04:50:52.098836 21144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:52.099946 21156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b0a8c9e26f4497981a9dad97956d6a0. Latest consensus state: current_term: 1 leader_uuid: "5b0a8c9e26f4497981a9dad97956d6a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0a8c9e26f4497981a9dad97956d6a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33011 } } }
I20260924 04:50:52.100054 21156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:52.100370 21155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b0a8c9e26f4497981a9dad97956d6a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0a8c9e26f4497981a9dad97956d6a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33011 } } }
I20260924 04:50:52.100488 21155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:52.100420 21163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:52.103885 21163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:52.109884 21163 catalog_manager.cc:1347] Generated new cluster ID: b1617c0553c642618455e803cd7c0efa
I20260924 04:50:52.109970 21163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:52.123395 21163 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:52.124436 21163 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:52.129644 21163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b0a8c9e26f4497981a9dad97956d6a0: Generated new TSK 0
I20260924 04:50:52.130580 21163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:50:52.145696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 21076
2026-09-24T04:50:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:50:52 dist-test-slave-hvcl krb5kdc[21175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:50:52 dist-test-slave-hvcl krb5kdc[21175](info): set up 2 sockets
Sep 24 04:50:52 dist-test-slave-hvcl krb5kdc[21175](info): commencing operation
krb5kdc: starting...
W20260924 04:50:53.098878 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:50:53 dist-test-slave-hvcl krb5kdc[21175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:50:53.218601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43363
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:53.329123 21188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:53.329437 21188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:53.329501 21188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:53.333762 21188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:53.333854 21188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:53.333884 21188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:53.333904 21188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:53.333925 21188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:53.339541 21188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:53.340720 21188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:53.341764 21188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:50:53.350562 21188 server_base.cc:1034] running on GCE node
W20260924 04:50:53.350550 21196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:53.350525 21193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:53.350525 21194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:53.351414 21188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:53.352526 21188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:53.353685 21188 hybrid_clock.cc:648] HybridClock initialized: now 1790225453353663 us; error 36 us; skew 500 ppm
Sep 24 04:50:53 dist-test-slave-hvcl krb5kdc[21175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:53.356392 21188 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:50:53.357338 21188 webserver.cc:460] Webserver started at http://127.0.0.1:37705/ using document root <none> and password file <none>
I20260924 04:50:53.357859 21188 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:53.357916 21188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:53.358076 21188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:53.360132 21188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "5f12f5b0677142c6b0c8d771cf795f89"
format_stamp: "Formatted at 2026-09-24 04:50:53 on dist-test-slave-hvcl"
server_key: "a1a57593bbe92aa3cd6caf5428e5818a"
server_key_iv: "825b06ae1ebcab1d424536f7623b4669"
server_key_version: "encryptionkey@0"
I20260924 04:50:53.360644 21188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f12f5b0677142c6b0c8d771cf795f89"
format_stamp: "Formatted at 2026-09-24 04:50:53 on dist-test-slave-hvcl"
server_key: "a1a57593bbe92aa3cd6caf5428e5818a"
server_key_iv: "825b06ae1ebcab1d424536f7623b4669"
server_key_version: "encryptionkey@0"
I20260924 04:50:53.364813 21188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:50:53.367219 21203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:53.368477 21188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:50:53.368592 21188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "5f12f5b0677142c6b0c8d771cf795f89"
format_stamp: "Formatted at 2026-09-24 04:50:53 on dist-test-slave-hvcl"
server_key: "a1a57593bbe92aa3cd6caf5428e5818a"
server_key_iv: "825b06ae1ebcab1d424536f7623b4669"
server_key_version: "encryptionkey@0"
I20260924 04:50:53.368748 21188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:53.373247 21188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:53.376406 21188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:53.376556 21188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:53.383265 21188 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43363
I20260924 04:50:53.383285 21255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43363 every 8 connection(s)
I20260924 04:50:53.384192 21188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:53.384963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 21188
I20260924 04:50:53.385080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:53.385339 392 external_mini_cluster.cc:1428] Setting key 8b8f5fb991c30089e746857e02cfaba0
I20260924 04:50:53.387306 21256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:53.390903 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:53.386729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35951 (local address 127.0.0.1:43363)
0924 04:50:53.387155 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:53.387174 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:53.387217 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:50:53.388055 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:53.388078 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:53.388443 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:53.388830 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:53.389475 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:53.389480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:53.389496 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:53.389505 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:53.389558 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:53.389583 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:53.389723 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:53.389803 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 04:50:53.389962 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":100,"threads_started":1}
I20260924 04:50:53.393849 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:50:53.394214 21256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89: Bootstrap starting.
I20260924 04:50:53.396682 21256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:53.397413 21256 log.cc:824] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:53.397552 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:53.395785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53285 (local address 127.0.0.1:43363)
0924 04:50:53.395920 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:53.395925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:53.396087 (+ 162us) server_negotiation.cc:407] Connection header received
0924 04:50:53.396215 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:53.396217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:53.396248 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:53.396308 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:53.397069 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:53.397073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:53.397076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:53.397087 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:53.397115 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:53.397150 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:53.397227 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:53.397332 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 04:50:53.397369 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:50:53.399534 21256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89: No bootstrap required, opened a new log
I20260924 04:50:53.403081 21256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f12f5b0677142c6b0c8d771cf795f89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260924 04:50:53.403343 21256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:53.403393 21256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f12f5b0677142c6b0c8d771cf795f89, State: Initialized, Role: FOLLOWER
I20260924 04:50:53.403852 21256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f12f5b0677142c6b0c8d771cf795f89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260924 04:50:53.404021 21256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:53.404094 21256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:53.404197 21256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:53.405231 21256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f12f5b0677142c6b0c8d771cf795f89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260924 04:50:53.405570 21256 leader_election.cc:302] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f12f5b0677142c6b0c8d771cf795f89; no voters:
I20260924 04:50:53.405879 21256 leader_election.cc:288] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:53.405982 21266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:53.406193 21266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [term 1 LEADER]: Becoming Leader. State: Replica: 5f12f5b0677142c6b0c8d771cf795f89, State: Running, Role: LEADER
I20260924 04:50:53.406510 21266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f12f5b0677142c6b0c8d771cf795f89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } }
I20260924 04:50:53.407109 21256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:53.408136 21268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f12f5b0677142c6b0c8d771cf795f89. Latest consensus state: current_term: 1 leader_uuid: "5f12f5b0677142c6b0c8d771cf795f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f12f5b0677142c6b0c8d771cf795f89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } } }
I20260924 04:50:53.408252 21268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:53.408505 21267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f12f5b0677142c6b0c8d771cf795f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f12f5b0677142c6b0c8d771cf795f89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43363 } } }
I20260924 04:50:53.408578 21267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:53.408623 21275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:53.411806 21275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:53.418496 21275 catalog_manager.cc:1347] Generated new cluster ID: 79645a011d2b44ed971f6ae1eec7b2a9
I20260924 04:50:53.418584 21275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:53.458177 21275 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:53.458525 21275 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:53.465231 21275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f12f5b0677142c6b0c8d771cf795f89: Generated new TSK 0
I20260924 04:50:53.465966 21275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:50:53.479560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 21188
2026-09-24T04:50:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:50:53 dist-test-slave-hvcl krb5kdc[21287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:50:53 dist-test-slave-hvcl krb5kdc[21287](info): set up 2 sockets
Sep 24 04:50:53 dist-test-slave-hvcl krb5kdc[21287](info): commencing operation
krb5kdc: starting...
W20260924 04:50:54.439038 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:50:54 dist-test-slave-hvcl krb5kdc[21287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:50:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:50:54.558702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:54.670413 21300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:54.670707 21300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:54.670751 21300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:54.675079 21300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:54.675151 21300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:54.675169 21300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:54.675187 21300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:54.675201 21300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:54.680835 21300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:54.682019 21300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:54.683117 21300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:54.692023 21308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:54.691923 21305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:54.692184 21300 server_base.cc:1034] running on GCE node
W20260924 04:50:54.691919 21306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:54.692859 21300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:54.694020 21300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:54.695178 21300 hybrid_clock.cc:648] HybridClock initialized: now 1790225454695140 us; error 50 us; skew 500 ppm
Sep 24 04:50:54 dist-test-slave-hvcl krb5kdc[21287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:54.698083 21300 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:50:54.699113 21300 webserver.cc:460] Webserver started at http://127.0.0.1:41015/ using document root <none> and password file <none>
I20260924 04:50:54.699679 21300 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:54.699749 21300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:54.699954 21300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:54.702111 21300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "11801dec883c4df499f1221b3cba7a24"
format_stamp: "Formatted at 2026-09-24 04:50:54 on dist-test-slave-hvcl"
server_key: "e69ece96e5de5c1ba64515ce4e66c03b"
server_key_iv: "f726f108802fe70c5ce268e0d9f63ec4"
server_key_version: "encryptionkey@0"
I20260924 04:50:54.702667 21300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "11801dec883c4df499f1221b3cba7a24"
format_stamp: "Formatted at 2026-09-24 04:50:54 on dist-test-slave-hvcl"
server_key: "e69ece96e5de5c1ba64515ce4e66c03b"
server_key_iv: "f726f108802fe70c5ce268e0d9f63ec4"
server_key_version: "encryptionkey@0"
I20260924 04:50:54.706552 21300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 04:50:54.709301 21315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:54.710618 21300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 04:50:54.710748 21300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "11801dec883c4df499f1221b3cba7a24"
format_stamp: "Formatted at 2026-09-24 04:50:54 on dist-test-slave-hvcl"
server_key: "e69ece96e5de5c1ba64515ce4e66c03b"
server_key_iv: "f726f108802fe70c5ce268e0d9f63ec4"
server_key_version: "encryptionkey@0"
I20260924 04:50:54.710868 21300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:54.727975 21300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:54.731086 21300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:54.731276 21300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:54.738193 21300 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46529
I20260924 04:50:54.738240 21367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46529 every 8 connection(s)
I20260924 04:50:54.739234 21300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:54.742085 21368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:54.745154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 21300
I20260924 04:50:54.745298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:54.745571 392 external_mini_cluster.cc:1428] Setting key ccb4e4bccff476318c6f3fe4644cea11
I20260924 04:50:54.748572 21368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24: Bootstrap starting.
I20260924 04:50:54.751189 21368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:54.752161 21368 log.cc:824] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:54.754580 21368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24: No bootstrap required, opened a new log
Sep 24 04:50:54 dist-test-slave-hvcl krb5kdc[21287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:50:54.758697 21368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11801dec883c4df499f1221b3cba7a24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46529 } }
I20260924 04:50:54.759052 21368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:54.759119 21368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11801dec883c4df499f1221b3cba7a24, State: Initialized, Role: FOLLOWER
I20260924 04:50:54.759666 21368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11801dec883c4df499f1221b3cba7a24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46529 } }
I20260924 04:50:54.759842 21368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:54.759958 21368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:54.760069 21368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:54.761057 21368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11801dec883c4df499f1221b3cba7a24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46529 } }
I20260924 04:50:54.761410 21368 leader_election.cc:302] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11801dec883c4df499f1221b3cba7a24; no voters:
I20260924 04:50:54.761722 21368 leader_election.cc:288] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:54.761822 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:54.747024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.54:48439 (local address 127.0.0.1:46529)
0924 04:50:54.747460 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:54.747480 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:54.747517 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:50:54.748250 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:54.748278 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:54.748693 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:54.749010 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:54.749912 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:54.750921 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:54.752132 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:54.752726 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:54.752905 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:54.755610 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:54.755617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:54.755643 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:54.755678 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:54.758133 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:54.758703 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:54.758709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:54.758715 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:54.758866 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:54.759237 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:54.759241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:54.759243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:54.759798 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:54.760046 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:54.760673 (+ 627us) server_negotiation.cc:299] Negotiation successful
0924 04:50:54.760894 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":113,"threads_started":1}
I20260924 04:50:54.761826 21373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:54.762025 21373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [term 1 LEADER]: Becoming Leader. State: Replica: 11801dec883c4df499f1221b3cba7a24, State: Running, Role: LEADER
I20260924 04:50:54.762327 21373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11801dec883c4df499f1221b3cba7a24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46529 } }
I20260924 04:50:54.763130 21368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:54.763664 21375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11801dec883c4df499f1221b3cba7a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11801dec883c4df499f1221b3cba7a24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46529 } } }
I20260924 04:50:54.763789 21375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:54.764065 21374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11801dec883c4df499f1221b3cba7a24. Latest consensus state: current_term: 1 leader_uuid: "11801dec883c4df499f1221b3cba7a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11801dec883c4df499f1221b3cba7a24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46529 } } }
I20260924 04:50:54.764207 21374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:54.764513 21377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:54.765398 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:50:54.768366 21377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:54.774773 21377 catalog_manager.cc:1347] Generated new cluster ID: c33f9db592c14d78b44b1c4d3eab560a
I20260924 04:50:54.774855 21377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:54.779546 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:54.767578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.54:51173 (local address 127.0.0.1:46529)
0924 04:50:54.767704 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:54.767709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:54.770741 (+ 3032us) server_negotiation.cc:407] Connection header received
0924 04:50:54.770883 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:54.770886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:54.770942 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:54.771044 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:54.772095 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:54.773331 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:54.774467 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:54.774818 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:54.774914 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:54.776475 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:54.776480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:54.776483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:54.776509 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:54.777811 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:54.778299 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:54.778302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:54.778303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:54.778377 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:54.778681 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:54.778684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:54.778687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:54.778862 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:54.779034 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:54.779290 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 04:50:54.779381 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:50:54.782899 21377 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:54.784154 21377 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:54.794114 21377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11801dec883c4df499f1221b3cba7a24: Generated new TSK 0
I20260924 04:50:54.794852 21377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:50:54.802353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 21300
W20260924 04:50:54.807121 21285 connection.cc:535] client connection to 127.0.0.1:46529 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46529): Connection reset by peer (error 104)
2026-09-24T04:50:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:50:54 dist-test-slave-hvcl krb5kdc[21399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:50:54 dist-test-slave-hvcl krb5kdc[21399](info): set up 2 sockets
Sep 24 04:50:54 dist-test-slave-hvcl krb5kdc[21399](info): commencing operation
krb5kdc: starting...
W20260924 04:50:55.762952 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:50:55 dist-test-slave-hvcl krb5kdc[21399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:50:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:50:55.886729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:55.998737 21412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:55.999089 21412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:55.999168 21412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:56.003407 21412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:56.003487 21412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:56.003516 21412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:56.003537 21412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:56.003558 21412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:56.009200 21412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:56.010375 21412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:56.011488 21412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:56.020473 21420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:56.020526 21417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:56.020589 21412 server_base.cc:1034] running on GCE node
W20260924 04:50:56.020498 21418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:56.021328 21412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:56.022547 21412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:56.023732 21412 hybrid_clock.cc:648] HybridClock initialized: now 1790225456023715 us; error 31 us; skew 500 ppm
Sep 24 04:50:56 dist-test-slave-hvcl krb5kdc[21399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:56.032052 21412 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:50:56.033140 21412 webserver.cc:460] Webserver started at http://127.0.0.1:43063/ using document root <none> and password file <none>
I20260924 04:50:56.033715 21412 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:56.033795 21412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:56.033980 21412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:56.036252 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "cd61a25751204119820d2c7554d1f310"
format_stamp: "Formatted at 2026-09-24 04:50:56 on dist-test-slave-hvcl"
server_key: "be87ecfaa4a7e065159e77331c783c62"
server_key_iv: "07dcdd19ea7832ac4c7a59844098e750"
server_key_version: "encryptionkey@0"
I20260924 04:50:56.036818 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd61a25751204119820d2c7554d1f310"
format_stamp: "Formatted at 2026-09-24 04:50:56 on dist-test-slave-hvcl"
server_key: "be87ecfaa4a7e065159e77331c783c62"
server_key_iv: "07dcdd19ea7832ac4c7a59844098e750"
server_key_version: "encryptionkey@0"
I20260924 04:50:56.040874 21412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:50:56.043509 21427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:56.044899 21412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:50:56.045018 21412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "cd61a25751204119820d2c7554d1f310"
format_stamp: "Formatted at 2026-09-24 04:50:56 on dist-test-slave-hvcl"
server_key: "be87ecfaa4a7e065159e77331c783c62"
server_key_iv: "07dcdd19ea7832ac4c7a59844098e750"
server_key_version: "encryptionkey@0"
I20260924 04:50:56.045126 21412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:56.071967 21412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:56.075729 21412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:56.075908 21412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:56.082808 21412 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39527
I20260924 04:50:56.082818 21479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39527 every 8 connection(s)
I20260924 04:50:56.083864 21412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:56.086596 21480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:56.092381 21480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310: Bootstrap starting.
I20260924 04:50:56.093662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 21412
I20260924 04:50:56.093789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:56.094030 392 external_mini_cluster.cc:1428] Setting key 94adc6d08e8dca4f3fb45d1936521648
I20260924 04:50:56.095081 21480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:56.095935 21480 log.cc:824] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:56.097782 21480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310: No bootstrap required, opened a new log
I20260924 04:50:56.101373 21480 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd61a25751204119820d2c7554d1f310" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39527 } }
I20260924 04:50:56.101606 21480 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:56.101649 21480 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd61a25751204119820d2c7554d1f310, State: Initialized, Role: FOLLOWER
I20260924 04:50:56.102123 21480 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd61a25751204119820d2c7554d1f310" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39527 } }
I20260924 04:50:56.102283 21480 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:56.102355 21480 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:56.102445 21480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:56.103200 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:56.095208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.54:42045 (local address 127.0.0.1:39527)
0924 04:50:56.095668 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:56.095684 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:56.095722 (+ 38us) server_negotiation.cc:407] Connection header received
0924 04:50:56.096430 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:56.096452 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:56.096812 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:56.097155 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:56.097581 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:56.098583 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:56.099525 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:56.100030 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:56.100189 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:56.101137 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:56.101142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:56.101156 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:56.101576 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:56.101649 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:56.101681 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:56.101805 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:56.102035 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 04:50:56.102175 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":144,"threads_started":1}
I20260924 04:50:56.103637 21480 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd61a25751204119820d2c7554d1f310" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39527 } }
I20260924 04:50:56.103955 21480 leader_election.cc:302] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd61a25751204119820d2c7554d1f310; no voters:
I20260924 04:50:56.104254 21480 leader_election.cc:288] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:56.104377 21485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:56.104588 21485 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [term 1 LEADER]: Becoming Leader. State: Replica: cd61a25751204119820d2c7554d1f310, State: Running, Role: LEADER
I20260924 04:50:56.104931 21485 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd61a25751204119820d2c7554d1f310" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39527 } }
I20260924 04:50:56.105530 21480 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:56.106619 21486 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd61a25751204119820d2c7554d1f310" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd61a25751204119820d2c7554d1f310" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39527 } } }
I20260924 04:50:56.106742 21486 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:56.107070 21487 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd61a25751204119820d2c7554d1f310. Latest consensus state: current_term: 1 leader_uuid: "cd61a25751204119820d2c7554d1f310" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd61a25751204119820d2c7554d1f310" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39527 } } }
I20260924 04:50:56.107151 21487 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:56.107169 21494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:56.107470 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:50:56.110385 21494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:56.115723 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:56.109583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.54:52571 (local address 127.0.0.1:39527)
0924 04:50:56.109706 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:56.109710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:56.111303 (+ 1593us) server_negotiation.cc:407] Connection header received
0924 04:50:56.111439 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:56.111442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:56.111494 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:56.111563 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:56.112011 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:56.112630 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:56.113830 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:56.114164 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:56.114267 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:56.115206 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:56.115210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:56.115212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:56.115227 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:56.115255 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:56.115268 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:56.115352 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:56.115533 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 04:50:56.115571 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 04:50:56.116464 21494 catalog_manager.cc:1347] Generated new cluster ID: 44eb022d0bca47a6b0bfea5dbf1132b1
I20260924 04:50:56.116590 21494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:56.125355 21494 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:56.126710 21494 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:56.136160 21494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd61a25751204119820d2c7554d1f310: Generated new TSK 0
I20260924 04:50:56.136940 21494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:50:56.143103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 21412
W20260924 04:50:56.147907 21397 connection.cc:535] client connection to 127.0.0.1:39527 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39527): Connection reset by peer (error 104)
2026-09-24T04:50:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:50:56 dist-test-slave-hvcl krb5kdc[21511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:50:56 dist-test-slave-hvcl krb5kdc[21511](info): set up 2 sockets
Sep 24 04:50:56 dist-test-slave-hvcl krb5kdc[21511](info): commencing operation
krb5kdc: starting...
W20260924 04:50:57.099156 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:50:57 dist-test-slave-hvcl krb5kdc[21511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:50:57.218024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:57.329402 21524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:57.329694 21524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:57.329736 21524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:57.333843 21524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:57.333923 21524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:57.333941 21524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:57.333957 21524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:57.333971 21524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:57.339540 21524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:57.340638 21524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:57.341658 21524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:57.350749 21532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:57.350841 21530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:57.350895 21529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:57.351001 21524 server_base.cc:1034] running on GCE node
I20260924 04:50:57.351773 21524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:57.352898 21524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:57.354065 21524 hybrid_clock.cc:648] HybridClock initialized: now 1790225457354025 us; error 52 us; skew 500 ppm
Sep 24 04:50:57 dist-test-slave-hvcl krb5kdc[21511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:57.356768 21524 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:50:57.357712 21524 webserver.cc:460] Webserver started at http://127.0.0.1:41779/ using document root <none> and password file <none>
I20260924 04:50:57.358263 21524 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:57.358327 21524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:57.358517 21524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:57.360637 21524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "b216ad7513bb4cd7882e89f698c61ea2"
format_stamp: "Formatted at 2026-09-24 04:50:57 on dist-test-slave-hvcl"
server_key: "bd9d88481a1bdb98d23990ed999cc795"
server_key_iv: "7b71ded5e818d760cd8b4bd911721f0a"
server_key_version: "encryptionkey@0"
I20260924 04:50:57.361222 21524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "b216ad7513bb4cd7882e89f698c61ea2"
format_stamp: "Formatted at 2026-09-24 04:50:57 on dist-test-slave-hvcl"
server_key: "bd9d88481a1bdb98d23990ed999cc795"
server_key_iv: "7b71ded5e818d760cd8b4bd911721f0a"
server_key_version: "encryptionkey@0"
I20260924 04:50:57.365280 21524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:50:57.367722 21539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:57.368988 21524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:50:57.369122 21524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "b216ad7513bb4cd7882e89f698c61ea2"
format_stamp: "Formatted at 2026-09-24 04:50:57 on dist-test-slave-hvcl"
server_key: "bd9d88481a1bdb98d23990ed999cc795"
server_key_iv: "7b71ded5e818d760cd8b4bd911721f0a"
server_key_version: "encryptionkey@0"
I20260924 04:50:57.369249 21524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:57.374014 21524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:57.377353 21524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:57.377535 21524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:57.384423 21524 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33519
I20260924 04:50:57.384449 21591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33519 every 8 connection(s)
I20260924 04:50:57.385370 21524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:57.388137 21592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:57.393630 21592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2: Bootstrap starting.
I20260924 04:50:57.394341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 21524
I20260924 04:50:57.394483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:57.394742 392 external_mini_cluster.cc:1428] Setting key 97b7a2623031f1b2f813bac7b3b6edbf
I20260924 04:50:57.396406 21592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:57.397338 21592 log.cc:824] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:57.399662 21592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2: No bootstrap required, opened a new log
I20260924 04:50:57.400452 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:57.395950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.54:40931 (local address 127.0.0.1:33519)
0924 04:50:57.396442 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:57.396461 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:57.396506 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:50:57.397379 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:57.397404 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:57.397782 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:57.398107 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:57.398902 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:57.398908 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:57.398923 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:57.398935 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:57.399030 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:57.399065 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:57.399177 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:57.399344 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 04:50:57.399509 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":144,"threads_started":1}
I20260924 04:50:57.403594 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:50:57.403928 21592 raft_consensus.cc:348] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b216ad7513bb4cd7882e89f698c61ea2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33519 } }
I20260924 04:50:57.404160 21592 raft_consensus.cc:374] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:57.404202 21592 raft_consensus.cc:729] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b216ad7513bb4cd7882e89f698c61ea2, State: Initialized, Role: FOLLOWER
I20260924 04:50:57.404609 21592 consensus_queue.cc:260] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b216ad7513bb4cd7882e89f698c61ea2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33519 } }
I20260924 04:50:57.404735 21592 raft_consensus.cc:388] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:57.404799 21592 raft_consensus.cc:482] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:57.404894 21592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:57.406020 21592 raft_consensus.cc:504] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b216ad7513bb4cd7882e89f698c61ea2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33519 } }
I20260924 04:50:57.406402 21592 leader_election.cc:302] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b216ad7513bb4cd7882e89f698c61ea2; no voters:
I20260924 04:50:57.406726 21592 leader_election.cc:288] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:57.406868 21602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:57.407083 21602 raft_consensus.cc:686] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [term 1 LEADER]: Becoming Leader. State: Replica: b216ad7513bb4cd7882e89f698c61ea2, State: Running, Role: LEADER
I20260924 04:50:57.407433 21602 consensus_queue.cc:237] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b216ad7513bb4cd7882e89f698c61ea2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33519 } }
I20260924 04:50:57.407586 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:57.405664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.54:56781 (local address 127.0.0.1:33519)
0924 04:50:57.405799 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:57.405802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:57.406023 (+ 221us) server_negotiation.cc:407] Connection header received
0924 04:50:57.406157 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:57.406159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:57.406198 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:57.406270 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:57.407090 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:57.407093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:57.407096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:50:57.407108 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:57.407136 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:50:57.407149 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:57.407204 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:57.407358 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 04:50:57.407400 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:50:57.408088 21592 sys_catalog.cc:564] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:57.408855 21602 sys_catalog.cc:455] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b216ad7513bb4cd7882e89f698c61ea2. Latest consensus state: current_term: 1 leader_uuid: "b216ad7513bb4cd7882e89f698c61ea2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b216ad7513bb4cd7882e89f698c61ea2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33519 } } }
I20260924 04:50:57.409022 21602 sys_catalog.cc:458] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:57.409236 21604 sys_catalog.cc:455] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b216ad7513bb4cd7882e89f698c61ea2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b216ad7513bb4cd7882e89f698c61ea2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33519 } } }
I20260924 04:50:57.409341 21604 sys_catalog.cc:458] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2 [sys.catalog]: This master's current role is: LEADER
W20260924 04:50:57.413257 21617 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:50:57.413357 21617 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:50:57.413478 21611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:57.414203 21611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:57.419500 21611 catalog_manager.cc:1347] Generated new cluster ID: d78572f03c11469e9bcbae9568573463
I20260924 04:50:57.419581 21611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:57.438130 21611 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:57.438485 21611 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:57.444562 21611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b216ad7513bb4cd7882e89f698c61ea2: Generated new TSK 0
I20260924 04:50:57.445379 21611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:50:57.453940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 21524
2026-09-24T04:50:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:50:57 dist-test-slave-hvcl krb5kdc[21623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:50:57 dist-test-slave-hvcl krb5kdc[21623](info): set up 2 sockets
Sep 24 04:50:57 dist-test-slave-hvcl krb5kdc[21623](info): commencing operation
krb5kdc: starting...
W20260924 04:50:58.423108 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:50:58 dist-test-slave-hvcl krb5kdc[21623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:50:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:50:58.541997 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:50:58.651961 21636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:50:58.652276 21636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:50:58.652338 21636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:50:58.656526 21636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:50:58.656610 21636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:50:58.656636 21636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:50:58.656667 21636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:50:58.656697 21636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:50:58.662178 21636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:50:58.663338 21636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:50:58.664371 21636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:50:58.673087 21644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:58.673079 21641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:50:58.673079 21642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:50:58.673187 21636 server_base.cc:1034] running on GCE node
I20260924 04:50:58.673938 21636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:50:58.675225 21636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:50:58.676379 21636 hybrid_clock.cc:648] HybridClock initialized: now 1790225458676354 us; error 42 us; skew 500 ppm
Sep 24 04:50:58 dist-test-slave-hvcl krb5kdc[21623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:50:58.679175 21636 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:50:58.680081 21636 webserver.cc:460] Webserver started at http://127.0.0.1:36877/ using document root <none> and password file <none>
I20260924 04:50:58.680636 21636 fs_manager.cc:359] Metadata directory not provided
I20260924 04:50:58.680698 21636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:50:58.680861 21636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:50:58.682863 21636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "2e3ffd00c3f34544bb4ffd87723f62dc"
format_stamp: "Formatted at 2026-09-24 04:50:58 on dist-test-slave-hvcl"
server_key: "774754903cbf1e814d97265576758e2b"
server_key_iv: "6572fd50f19ce1e96b696131ccd3652b"
server_key_version: "encryptionkey@0"
I20260924 04:50:58.683427 21636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e3ffd00c3f34544bb4ffd87723f62dc"
format_stamp: "Formatted at 2026-09-24 04:50:58 on dist-test-slave-hvcl"
server_key: "774754903cbf1e814d97265576758e2b"
server_key_iv: "6572fd50f19ce1e96b696131ccd3652b"
server_key_version: "encryptionkey@0"
I20260924 04:50:58.687232 21636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:50:58.689829 21651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:50:58.691238 21636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:50:58.691368 21636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "2e3ffd00c3f34544bb4ffd87723f62dc"
format_stamp: "Formatted at 2026-09-24 04:50:58 on dist-test-slave-hvcl"
server_key: "774754903cbf1e814d97265576758e2b"
server_key_iv: "6572fd50f19ce1e96b696131ccd3652b"
server_key_version: "encryptionkey@0"
I20260924 04:50:58.691483 21636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:50:58.710021 21636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:50:58.713181 21636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:50:58.713366 21636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:50:58.720233 21636 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40533
I20260924 04:50:58.720271 21703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40533 every 8 connection(s)
I20260924 04:50:58.721170 21636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:50:58.723968 21704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:50:58.728626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 21636
I20260924 04:50:58.728780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:50:58.729029 392 external_mini_cluster.cc:1428] Setting key 5d6d7eba169534ab67bd0c7f5c5fa401
I20260924 04:50:58.729580 21704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc: Bootstrap starting.
I20260924 04:50:58.732023 21704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc: Neither blocks nor log segments found. Creating new log.
I20260924 04:50:58.732756 21704 log.cc:824] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:50:58.734535 21704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc: No bootstrap required, opened a new log
Sep 24 04:50:58 dist-test-slave-hvcl krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:50:58.738250 21704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3ffd00c3f34544bb4ffd87723f62dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40533 } }
I20260924 04:50:58.738477 21704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:50:58.738543 21704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e3ffd00c3f34544bb4ffd87723f62dc, State: Initialized, Role: FOLLOWER
I20260924 04:50:58.739017 21704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3ffd00c3f34544bb4ffd87723f62dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40533 } }
I20260924 04:50:58.739151 21704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:50:58.739214 21704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:50:58.739315 21704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:50:58.740427 21704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3ffd00c3f34544bb4ffd87723f62dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40533 } }
I20260924 04:50:58.740765 21704 leader_election.cc:302] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e3ffd00c3f34544bb4ffd87723f62dc; no voters:
I20260924 04:50:58.741117 21704 leader_election.cc:288] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:50:58.741264 21709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:50:58.741503 21709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [term 1 LEADER]: Becoming Leader. State: Replica: 2e3ffd00c3f34544bb4ffd87723f62dc, State: Running, Role: LEADER
I20260924 04:50:58.741854 21709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3ffd00c3f34544bb4ffd87723f62dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40533 } }
I20260924 04:50:58.742446 21704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:50:58.743630 21711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e3ffd00c3f34544bb4ffd87723f62dc. Latest consensus state: current_term: 1 leader_uuid: "2e3ffd00c3f34544bb4ffd87723f62dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3ffd00c3f34544bb4ffd87723f62dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40533 } } }
I20260924 04:50:58.743757 21711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:58.743814 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:58.730379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.54:48125 (local address 127.0.0.1:40533)
0924 04:50:58.730796 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:58.730815 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:58.730858 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:50:58.731542 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:58.731560 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:58.731936 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:58.732273 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:58.733072 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:58.734048 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:58.734922 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:58.735433 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:58.735586 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:58.738213 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:58.738218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:58.738231 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:58.738258 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:58.740180 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:58.740809 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:58.740816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:58.740822 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:58.740962 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:58.741444 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:58.741447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:58.741449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:58.741973 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:58.742186 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:58.742523 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 04:50:58.742757 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":129,"threads_started":1}
I20260924 04:50:58.744055 21710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e3ffd00c3f34544bb4ffd87723f62dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3ffd00c3f34544bb4ffd87723f62dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40533 } } }
I20260924 04:50:58.744139 21710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc [sys.catalog]: This master's current role is: LEADER
I20260924 04:50:58.744156 21718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:50:58.747161 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:50:58.747496 21718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:50:58.753216 21718 catalog_manager.cc:1347] Generated new cluster ID: e0028515d56e4f80bf353804a66f9cbd
I20260924 04:50:58.753283 21718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:50:58.758905 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:50:58.749437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.54:44943 (local address 127.0.0.1:40533)
0924 04:50:58.749547 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:50:58.749551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:50:58.751277 (+ 1726us) server_negotiation.cc:407] Connection header received
0924 04:50:58.751479 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:50:58.751483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:50:58.751542 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:50:58.751644 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:50:58.752570 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:58.753316 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:58.754141 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:50:58.754392 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:50:58.754478 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:50:58.755792 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:50:58.755796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:50:58.755799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:50:58.755825 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:50:58.757246 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:58.757697 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:58.757700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:58.757702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:58.757780 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:50:58.758068 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:50:58.758071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:50:58.758073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:50:58.758255 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:50:58.758384 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 04:50:58.758634 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 04:50:58.758744 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:50:58.800292 21718 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:50:58.801326 21718 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:50:58.808916 21718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e3ffd00c3f34544bb4ffd87723f62dc: Generated new TSK 0
I20260924 04:50:58.809636 21718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:50:58.819065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 21636
W20260924 04:50:58.823746 21621 connection.cc:535] client connection to 127.0.0.1:40533 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40533): Connection reset by peer (error 104)
2026-09-24T04:50:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:50:58 dist-test-slave-hvcl krb5kdc[21735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:50:58 dist-test-slave-hvcl krb5kdc[21735](info): set up 2 sockets
Sep 24 04:50:58 dist-test-slave-hvcl krb5kdc[21735](info): commencing operation
krb5kdc: starting...
W20260924 04:50:59.778910 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:50:59 dist-test-slave-hvcl krb5kdc[21735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:50:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:50:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:50:59.899222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:51:00.008136 21748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:51:00.008425 21748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:51:00.008468 21748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:51:00.012632 21748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:51:00.012712 21748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:51:00.012730 21748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:51:00.012745 21748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:51:00.012766 21748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:51:00.018190 21748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43549
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:51:00.019337 21748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:51:00.020372 21748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:51:00.029021 21754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:00.029007 21756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:00.029011 21753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:00.029368 21748 server_base.cc:1034] running on GCE node
I20260924 04:51:00.030067 21748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:51:00.031317 21748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:51:00.032510 21748 hybrid_clock.cc:648] HybridClock initialized: now 1790225460032491 us; error 33 us; skew 500 ppm
Sep 24 04:51:00 dist-test-slave-hvcl krb5kdc[21735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:51:00.035454 21748 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:51:00.036432 21748 webserver.cc:460] Webserver started at http://127.0.0.1:39633/ using document root <none> and password file <none>
I20260924 04:51:00.036991 21748 fs_manager.cc:359] Metadata directory not provided
I20260924 04:51:00.037060 21748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:51:00.037293 21748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:51:00.039501 21748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "bedb534c91cf4feaab88283cc1a7ca23"
format_stamp: "Formatted at 2026-09-24 04:51:00 on dist-test-slave-hvcl"
server_key: "671c313848340870a4287ccaf462ef24"
server_key_iv: "ce364163a4fac8dfaec3d25369b72b4a"
server_key_version: "encryptionkey@0"
I20260924 04:51:00.040045 21748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "bedb534c91cf4feaab88283cc1a7ca23"
format_stamp: "Formatted at 2026-09-24 04:51:00 on dist-test-slave-hvcl"
server_key: "671c313848340870a4287ccaf462ef24"
server_key_iv: "ce364163a4fac8dfaec3d25369b72b4a"
server_key_version: "encryptionkey@0"
I20260924 04:51:00.043817 21748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:51:00.046657 21763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:51:00.047943 21748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:51:00.048060 21748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "bedb534c91cf4feaab88283cc1a7ca23"
format_stamp: "Formatted at 2026-09-24 04:51:00 on dist-test-slave-hvcl"
server_key: "671c313848340870a4287ccaf462ef24"
server_key_iv: "ce364163a4fac8dfaec3d25369b72b4a"
server_key_version: "encryptionkey@0"
I20260924 04:51:00.048161 21748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:51:00.077242 21748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:51:00.080617 21748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:51:00.080807 21748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:51:00.087637 21748 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43549
I20260924 04:51:00.087749 21815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43549 every 8 connection(s)
I20260924 04:51:00.088608 21748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:51:00.091390 21816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:51:00.095541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 21748
I20260924 04:51:00.095672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:51:00.095912 392 external_mini_cluster.cc:1428] Setting key 4d361b12621e225a8e0256e0de48c50e
I20260924 04:51:00.097048 21816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23: Bootstrap starting.
I20260924 04:51:00.099314 21816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23: Neither blocks nor log segments found. Creating new log.
I20260924 04:51:00.100028 21816 log.cc:824] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23: Log is configured to *not* fsync() on all Append() calls
I20260924 04:51:00.102013 21816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23: No bootstrap required, opened a new log
I20260924 04:51:00.105054 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:00.097317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.54:56231 (local address 127.0.0.1:43549)
0924 04:51:00.097720 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:00.097736 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:00.097771 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:51:00.098396 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:00.098411 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:00.098776 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:00.099094 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:00.099422 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:00.100437 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:00.101345 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:00.102001 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:00.102156 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:00.103048 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:00.103054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:00.103069 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:51:00.103563 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:00.103628 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:51:00.103657 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:00.103788 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:00.103956 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 04:51:00.104087 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260924 04:51:00.106300 21816 raft_consensus.cc:348] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedb534c91cf4feaab88283cc1a7ca23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20260924 04:51:00.106528 21816 raft_consensus.cc:374] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:51:00.106566 21816 raft_consensus.cc:729] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bedb534c91cf4feaab88283cc1a7ca23, State: Initialized, Role: FOLLOWER
I20260924 04:51:00.107033 21816 consensus_queue.cc:260] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedb534c91cf4feaab88283cc1a7ca23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20260924 04:51:00.107169 21816 raft_consensus.cc:388] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:51:00.107234 21816 raft_consensus.cc:482] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:51:00.107329 21816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:51:00.108132 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:51:00.108315 21816 raft_consensus.cc:504] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedb534c91cf4feaab88283cc1a7ca23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20260924 04:51:00.108634 21816 leader_election.cc:302] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bedb534c91cf4feaab88283cc1a7ca23; no voters:
I20260924 04:51:00.108934 21816 leader_election.cc:288] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:51:00.109030 21821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:51:00.109232 21821 raft_consensus.cc:686] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [term 1 LEADER]: Becoming Leader. State: Replica: bedb534c91cf4feaab88283cc1a7ca23, State: Running, Role: LEADER
I20260924 04:51:00.109562 21821 consensus_queue.cc:237] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedb534c91cf4feaab88283cc1a7ca23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20260924 04:51:00.110129 21816 sys_catalog.cc:564] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:51:00.110793 21824 sys_catalog.cc:455] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bedb534c91cf4feaab88283cc1a7ca23. Latest consensus state: current_term: 1 leader_uuid: "bedb534c91cf4feaab88283cc1a7ca23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedb534c91cf4feaab88283cc1a7ca23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } } }
I20260924 04:51:00.110917 21824 sys_catalog.cc:458] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:00.111191 21822 sys_catalog.cc:455] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bedb534c91cf4feaab88283cc1a7ca23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedb534c91cf4feaab88283cc1a7ca23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } } }
I20260924 04:51:00.111295 21822 sys_catalog.cc:458] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23 [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:00.111330 21835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:51:00.114478 21835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:51:00.115454 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:00.110239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.54:43739 (local address 127.0.0.1:43549)
0924 04:51:00.110348 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:00.110352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:00.110711 (+ 359us) server_negotiation.cc:407] Connection header received
0924 04:51:00.110893 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:00.110898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:00.110946 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:00.111020 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:00.111444 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:00.112251 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:00.113356 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:00.113670 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:00.113742 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:00.114769 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:00.114772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:00.114775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:51:00.114788 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:00.114813 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:51:00.114825 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:00.114882 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:00.115285 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 04:51:00.115322 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:51:00.120182 21835 catalog_manager.cc:1347] Generated new cluster ID: 9c947f95788a40959d0059e5b0f78cd1
I20260924 04:51:00.120247 21835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:51:00.136799 21835 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:51:00.138082 21835 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:51:00.149348 21835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bedb534c91cf4feaab88283cc1a7ca23: Generated new TSK 0
I20260924 04:51:00.150200 21835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:51:00.163811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 21748
W20260924 04:51:00.168792 21733 connection.cc:535] client connection to 127.0.0.1:43549 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43549): Connection reset by peer (error 104)
2026-09-24T04:51:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:51:00 dist-test-slave-hvcl krb5kdc[21847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:51:00 dist-test-slave-hvcl krb5kdc[21847](info): set up 2 sockets
Sep 24 04:51:00 dist-test-slave-hvcl krb5kdc[21847](info): commencing operation
krb5kdc: starting...
W20260924 04:51:01.115085 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:51:01 dist-test-slave-hvcl krb5kdc[21847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:51:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:51:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:51:01.234030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:51:01.343783 21860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:51:01.344070 21860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:51:01.344111 21860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:51:01.348157 21860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:51:01.348232 21860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:51:01.348256 21860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:51:01.348272 21860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:51:01.348285 21860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:51:01.353679 21860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34281
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:51:01.354751 21860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:51:01.355827 21860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:51:01.364154 21866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:01.364154 21865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:01.364166 21868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:01.364219 21860 server_base.cc:1034] running on GCE node
I20260924 04:51:01.365037 21860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:51:01.366133 21860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:51:01.367285 21860 hybrid_clock.cc:648] HybridClock initialized: now 1790225461367259 us; error 39 us; skew 500 ppm
Sep 24 04:51:01 dist-test-slave-hvcl krb5kdc[21847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:51:01.373930 21860 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:51:01.375087 21860 webserver.cc:460] Webserver started at http://127.0.0.1:39389/ using document root <none> and password file <none>
I20260924 04:51:01.375674 21860 fs_manager.cc:359] Metadata directory not provided
I20260924 04:51:01.375753 21860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:51:01.375962 21860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:51:01.378037 21860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "0b1004c5047c4973a675bca418692f1e"
format_stamp: "Formatted at 2026-09-24 04:51:01 on dist-test-slave-hvcl"
server_key: "b2d7824c6b66d616db9a4c8386089454"
server_key_iv: "024b49c6d8fb1fad719c6d355e9e100f"
server_key_version: "encryptionkey@0"
I20260924 04:51:01.378573 21860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b1004c5047c4973a675bca418692f1e"
format_stamp: "Formatted at 2026-09-24 04:51:01 on dist-test-slave-hvcl"
server_key: "b2d7824c6b66d616db9a4c8386089454"
server_key_iv: "024b49c6d8fb1fad719c6d355e9e100f"
server_key_version: "encryptionkey@0"
I20260924 04:51:01.382439 21860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:51:01.385058 21875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:51:01.386302 21860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:51:01.386420 21860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "0b1004c5047c4973a675bca418692f1e"
format_stamp: "Formatted at 2026-09-24 04:51:01 on dist-test-slave-hvcl"
server_key: "b2d7824c6b66d616db9a4c8386089454"
server_key_iv: "024b49c6d8fb1fad719c6d355e9e100f"
server_key_version: "encryptionkey@0"
I20260924 04:51:01.386521 21860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:51:01.391043 21860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:51:01.394289 21860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:51:01.394464 21860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:51:01.401221 21860 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34281
I20260924 04:51:01.401238 21927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34281 every 8 connection(s)
I20260924 04:51:01.402163 21860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:51:01.405264 21928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:51:01.410162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 21860
I20260924 04:51:01.410326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:51:01.410614 392 external_mini_cluster.cc:1428] Setting key 98fda866414cfc3cf1b066a9ac22be7e
I20260924 04:51:01.411389 21928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e: Bootstrap starting.
I20260924 04:51:01.413839 21928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e: Neither blocks nor log segments found. Creating new log.
I20260924 04:51:01.414676 21928 log.cc:824] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:51:01.416077 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:01.412116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.54:42963 (local address 127.0.0.1:34281)
0924 04:51:01.412582 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:01.412597 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:01.412631 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:51:01.413292 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:01.413311 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:01.413612 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:01.413932 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:01.414706 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:01.414711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:01.414723 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:51:01.414732 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:01.414788 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:51:01.414810 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:01.414908 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:01.415115 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 04:51:01.415313 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":146,"threads_started":1}
I20260924 04:51:01.416929 21928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e: No bootstrap required, opened a new log
I20260924 04:51:01.418906 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:51:01.420826 21928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1004c5047c4973a675bca418692f1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34281 } }
I20260924 04:51:01.421125 21928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:51:01.421190 21928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b1004c5047c4973a675bca418692f1e, State: Initialized, Role: FOLLOWER
I20260924 04:51:01.421631 21928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1004c5047c4973a675bca418692f1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34281 } }
I20260924 04:51:01.421800 21928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:51:01.421868 21928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:51:01.421970 21928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:51:01.422513 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:01.420853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.54:35275 (local address 127.0.0.1:34281)
0924 04:51:01.420970 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:01.420974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:01.421143 (+ 169us) server_negotiation.cc:407] Connection header received
0924 04:51:01.421326 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:01.421328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:01.421362 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:01.421431 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:01.422151 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:01.422155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:01.422157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:51:01.422168 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:01.422198 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:51:01.422210 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:01.422303 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:01.422356 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 04:51:01.422395 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:51:01.423120 21928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1004c5047c4973a675bca418692f1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34281 } }
I20260924 04:51:01.423477 21928 leader_election.cc:302] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b1004c5047c4973a675bca418692f1e; no voters:
I20260924 04:51:01.423766 21928 leader_election.cc:288] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:51:01.423867 21938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:51:01.424072 21938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [term 1 LEADER]: Becoming Leader. State: Replica: 0b1004c5047c4973a675bca418692f1e, State: Running, Role: LEADER
I20260924 04:51:01.424433 21938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1004c5047c4973a675bca418692f1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34281 } }
I20260924 04:51:01.424993 21928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:51:01.426087 21940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b1004c5047c4973a675bca418692f1e. Latest consensus state: current_term: 1 leader_uuid: "0b1004c5047c4973a675bca418692f1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1004c5047c4973a675bca418692f1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34281 } } }
I20260924 04:51:01.426088 21939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b1004c5047c4973a675bca418692f1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1004c5047c4973a675bca418692f1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34281 } } }
I20260924 04:51:01.426332 21939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:01.426326 21940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:01.427029 21948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:51:01.429750 21948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:51:01.434867 21948 catalog_manager.cc:1347] Generated new cluster ID: ad9c41098d4f40a6bd04bfa87c18e57d
I20260924 04:51:01.434945 21948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:51:01.446175 21948 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:51:01.446516 21948 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:51:01.460595 21948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b1004c5047c4973a675bca418692f1e: Generated new TSK 0
I20260924 04:51:01.461367 21948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:51:01.464097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 21860
2026-09-24T04:51:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:51:01 dist-test-slave-hvcl krb5kdc[21959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:51:01 dist-test-slave-hvcl krb5kdc[21959](info): set up 2 sockets
Sep 24 04:51:01 dist-test-slave-hvcl krb5kdc[21959](info): commencing operation
krb5kdc: starting...
W20260924 04:51:02.418995 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:51:02 dist-test-slave-hvcl krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:51:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:51:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:51:02.538040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36267
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:51:02.647867 21972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:51:02.648169 21972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:51:02.648211 21972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:51:02.652695 21972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:51:02.652778 21972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:51:02.652797 21972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:51:02.652813 21972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:51:02.652827 21972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:51:02.658308 21972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:51:02.659390 21972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:51:02.660352 21972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:51:02.668771 21977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:02.668777 21980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:02.668812 21978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:02.669070 21972 server_base.cc:1034] running on GCE node
I20260924 04:51:02.669624 21972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:51:02.670735 21972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:51:02.671902 21972 hybrid_clock.cc:648] HybridClock initialized: now 1790225462671864 us; error 50 us; skew 500 ppm
Sep 24 04:51:02 dist-test-slave-hvcl krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:51:02.674871 21972 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:51:02.675817 21972 webserver.cc:460] Webserver started at http://127.0.98.62:38147/ using document root <none> and password file <none>
I20260924 04:51:02.676366 21972 fs_manager.cc:359] Metadata directory not provided
I20260924 04:51:02.676431 21972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:51:02.676622 21972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:51:02.678642 21972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "5d7377c43d844f93950976668eef2544"
format_stamp: "Formatted at 2026-09-24 04:51:02 on dist-test-slave-hvcl"
server_key: "3ed919fc9f53b06bb74269d4805e06e6"
server_key_iv: "e2bdd0db956973a3585e5fcdd0a47a8a"
server_key_version: "encryptionkey@0"
I20260924 04:51:02.679206 21972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "5d7377c43d844f93950976668eef2544"
format_stamp: "Formatted at 2026-09-24 04:51:02 on dist-test-slave-hvcl"
server_key: "3ed919fc9f53b06bb74269d4805e06e6"
server_key_iv: "e2bdd0db956973a3585e5fcdd0a47a8a"
server_key_version: "encryptionkey@0"
I20260924 04:51:02.682803 21972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:51:02.685295 21987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:51:02.686575 21972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:51:02.686781 21972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "5d7377c43d844f93950976668eef2544"
format_stamp: "Formatted at 2026-09-24 04:51:02 on dist-test-slave-hvcl"
server_key: "3ed919fc9f53b06bb74269d4805e06e6"
server_key_iv: "e2bdd0db956973a3585e5fcdd0a47a8a"
server_key_version: "encryptionkey@0"
I20260924 04:51:02.686890 21972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:51:02.697606 21972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:51:02.700546 21972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:51:02.700723 21972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:51:02.706952 21972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36267
I20260924 04:51:02.707062 22039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36267 every 8 connection(s)
I20260924 04:51:02.707896 21972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:51:02.710616 22040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:51:02.714066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 21972
I20260924 04:51:02.714182 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:51:02.714419 392 external_mini_cluster.cc:1428] Setting key 14f333d6b5799a419d6843feaa742ccc
I20260924 04:51:02.716393 22040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544: Bootstrap starting.
I20260924 04:51:02.719023 22040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544: Neither blocks nor log segments found. Creating new log.
I20260924 04:51:02.719866 22040 log.cc:824] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544: Log is configured to *not* fsync() on all Append() calls
I20260924 04:51:02.721882 22040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544: No bootstrap required, opened a new log
Sep 24 04:51:02 dist-test-slave-hvcl krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:51:02.725723 22040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7377c43d844f93950976668eef2544" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36267 } }
I20260924 04:51:02.725948 22040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:51:02.725984 22040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d7377c43d844f93950976668eef2544, State: Initialized, Role: FOLLOWER
I20260924 04:51:02.726429 22040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7377c43d844f93950976668eef2544" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36267 } }
I20260924 04:51:02.726567 22040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:51:02.726624 22040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:51:02.726722 22040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:51:02.727609 22040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7377c43d844f93950976668eef2544" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36267 } }
I20260924 04:51:02.727939 22040 leader_election.cc:302] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d7377c43d844f93950976668eef2544; no voters:
I20260924 04:51:02.728268 22040 leader_election.cc:288] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:51:02.728353 22045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:51:02.728538 22045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [term 1 LEADER]: Becoming Leader. State: Replica: 5d7377c43d844f93950976668eef2544, State: Running, Role: LEADER
I20260924 04:51:02.728878 22045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7377c43d844f93950976668eef2544" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36267 } }
I20260924 04:51:02.729007 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:02.715691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41601 (local address 127.0.98.62:36267)
0924 04:51:02.716099 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:02.716117 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:02.716157 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:51:02.716955 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:02.716978 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:02.717388 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:02.717755 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:02.718422 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:02.719913 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:02.720830 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:02.721471 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:02.721622 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:02.724239 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:02.724244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:02.724258 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:02.724285 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:02.725914 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:02.726404 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:02.726409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:02.726413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:02.726548 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:02.726877 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:02.726880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:02.726882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:02.727312 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:02.727519 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:02.727826 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 04:51:02.728034 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":101,"threads_started":1}
I20260924 04:51:02.729529 22040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:51:02.730350 22046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d7377c43d844f93950976668eef2544. Latest consensus state: current_term: 1 leader_uuid: "5d7377c43d844f93950976668eef2544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7377c43d844f93950976668eef2544" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36267 } } }
I20260924 04:51:02.730466 22046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:02.730656 22047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d7377c43d844f93950976668eef2544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7377c43d844f93950976668eef2544" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36267 } } }
I20260924 04:51:02.730724 22047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544 [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:02.731515 22049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:51:02.732575 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:51:02.734597 22049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:51:02.740270 22049 catalog_manager.cc:1347] Generated new cluster ID: 2f99b45b2d914f3799ac5cceed03e8e0
I20260924 04:51:02.740336 22049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:51:02.745903 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:02.736613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35017 (local address 127.0.98.62:36267)
0924 04:51:02.736729 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:02.736734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:02.738571 (+ 1837us) server_negotiation.cc:407] Connection header received
0924 04:51:02.738700 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:02.738703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:02.738769 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:02.738868 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:02.739732 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:02.740451 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:02.741227 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:02.741482 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:02.741564 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:02.742882 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:02.742885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:02.742887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:02.742909 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:02.744263 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:02.744696 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:02.744700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:02.744702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:02.744777 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:02.745077 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:02.745080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:02.745082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:02.745249 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:02.745373 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:02.745647 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 04:51:02.745735 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:51:02.769052 22049 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:51:02.770223 22049 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:51:02.780179 22049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d7377c43d844f93950976668eef2544: Generated new TSK 0
I20260924 04:51:02.780963 22049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:51:02.792936 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 21972
W20260924 04:51:02.798233 21957 connection.cc:535] client connection to 127.0.98.62:36267 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36267): Connection reset by peer (error 104)
2026-09-24T04:51:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:51:02 dist-test-slave-hvcl krb5kdc[22071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:51:02 dist-test-slave-hvcl krb5kdc[22071](info): set up 2 sockets
Sep 24 04:51:02 dist-test-slave-hvcl krb5kdc[22071](info): commencing operation
krb5kdc: starting...
W20260924 04:51:03.755187 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:51:03 dist-test-slave-hvcl krb5kdc[22071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:51:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:51:03.872251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46231
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:51:03.982532 22084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:51:03.982846 22084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:51:03.982892 22084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:51:03.987179 22084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:51:03.987253 22084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:51:03.987272 22084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:51:03.987287 22084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:51:03.987305 22084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:51:03.992784 22084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46231
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:51:03.993880 22084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:51:03.994935 22084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:51:04.003595 22084 server_base.cc:1034] running on GCE node
W20260924 04:51:04.003468 22092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:04.003468 22089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:04.003472 22090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:04.004362 22084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:51:04.005515 22084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:51:04.006714 22084 hybrid_clock.cc:648] HybridClock initialized: now 1790225464006683 us; error 34 us; skew 500 ppm
Sep 24 04:51:04 dist-test-slave-hvcl krb5kdc[22071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:51:04.009621 22084 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:51:04.010569 22084 webserver.cc:460] Webserver started at http://127.0.98.62:43067/ using document root <none> and password file <none>
I20260924 04:51:04.011180 22084 fs_manager.cc:359] Metadata directory not provided
I20260924 04:51:04.011255 22084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:51:04.011471 22084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:51:04.013625 22084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "94c3e2c1f5994d558a7662928bf179ed"
format_stamp: "Formatted at 2026-09-24 04:51:04 on dist-test-slave-hvcl"
server_key: "dd73b86ebdc763af2c4320cd66aa1f7a"
server_key_iv: "cc67136ae8dd7d0dea84e935054944b5"
server_key_version: "encryptionkey@0"
I20260924 04:51:04.014189 22084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "94c3e2c1f5994d558a7662928bf179ed"
format_stamp: "Formatted at 2026-09-24 04:51:04 on dist-test-slave-hvcl"
server_key: "dd73b86ebdc763af2c4320cd66aa1f7a"
server_key_iv: "cc67136ae8dd7d0dea84e935054944b5"
server_key_version: "encryptionkey@0"
I20260924 04:51:04.017987 22084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 04:51:04.020617 22099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:51:04.022030 22084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:51:04.022163 22084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "94c3e2c1f5994d558a7662928bf179ed"
format_stamp: "Formatted at 2026-09-24 04:51:04 on dist-test-slave-hvcl"
server_key: "dd73b86ebdc763af2c4320cd66aa1f7a"
server_key_iv: "cc67136ae8dd7d0dea84e935054944b5"
server_key_version: "encryptionkey@0"
I20260924 04:51:04.022282 22084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:51:04.033330 22084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:51:04.036783 22084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:51:04.036980 22084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:51:04.044031 22084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46231
I20260924 04:51:04.044049 22151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46231 every 8 connection(s)
I20260924 04:51:04.045017 22084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:51:04.047982 22152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:51:04.048740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 22084
I20260924 04:51:04.048877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:51:04.049180 392 external_mini_cluster.cc:1428] Setting key f759924497ed498506690ae74c803550
I20260924 04:51:04.054468 22152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed: Bootstrap starting.
I20260924 04:51:04.057118 22152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed: Neither blocks nor log segments found. Creating new log.
I20260924 04:51:04.057945 22152 log.cc:824] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed: Log is configured to *not* fsync() on all Append() calls
I20260924 04:51:04.058571 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:04.050722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38891 (local address 127.0.98.62:46231)
0924 04:51:04.051195 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:04.051210 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:04.051246 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:51:04.051939 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:04.051959 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:04.052342 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:04.052632 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:04.052995 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:04.053978 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:04.055179 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:04.055651 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:04.055772 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:04.056737 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:04.056743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:04.056756 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:51:04.057154 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:04.057229 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:51:04.057261 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:04.057378 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:04.057563 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 04:51:04.057701 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":145,"threads_started":1}
I20260924 04:51:04.060048 22152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed: No bootstrap required, opened a new log
I20260924 04:51:04.061561 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:51:04.064031 22152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c3e2c1f5994d558a7662928bf179ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260924 04:51:04.064319 22152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:51:04.064379 22152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94c3e2c1f5994d558a7662928bf179ed, State: Initialized, Role: FOLLOWER
I20260924 04:51:04.064862 22152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c3e2c1f5994d558a7662928bf179ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260924 04:51:04.065034 22152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:51:04.065102 22152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:51:04.065199 22152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:51:04.066208 22152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c3e2c1f5994d558a7662928bf179ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260924 04:51:04.066607 22152 leader_election.cc:302] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94c3e2c1f5994d558a7662928bf179ed; no voters:
I20260924 04:51:04.067039 22152 leader_election.cc:288] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:51:04.067142 22162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:51:04.067406 22162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [term 1 LEADER]: Becoming Leader. State: Replica: 94c3e2c1f5994d558a7662928bf179ed, State: Running, Role: LEADER
I20260924 04:51:04.067772 22162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c3e2c1f5994d558a7662928bf179ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260924 04:51:04.067986 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:04.063503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47843 (local address 127.0.98.62:46231)
0924 04:51:04.063610 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:04.063614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:04.063745 (+ 131us) server_negotiation.cc:407] Connection header received
0924 04:51:04.063828 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:04.063831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:04.063873 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:04.063950 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:04.064158 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:04.064894 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:04.065852 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:04.066211 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:04.066318 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:04.067262 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:04.067266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:04.067269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:51:04.067283 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:04.067310 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:51:04.067323 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:04.067406 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:04.067814 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 04:51:04.067848 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:51:04.068421 22152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:51:04.069455 22163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94c3e2c1f5994d558a7662928bf179ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c3e2c1f5994d558a7662928bf179ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } } }
I20260924 04:51:04.069595 22163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:04.069455 22164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94c3e2c1f5994d558a7662928bf179ed. Latest consensus state: current_term: 1 leader_uuid: "94c3e2c1f5994d558a7662928bf179ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c3e2c1f5994d558a7662928bf179ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } } }
I20260924 04:51:04.069844 22164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:04.070185 22172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:51:04.073216 22172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:51:04.078613 22172 catalog_manager.cc:1347] Generated new cluster ID: 30d234aab4df45a6a2b12c733102ba8b
I20260924 04:51:04.078817 22172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:51:04.088793 22172 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:51:04.089712 22172 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:51:04.094512 22172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94c3e2c1f5994d558a7662928bf179ed: Generated new TSK 0
I20260924 04:51:04.095345 22172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:51:04.104127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 22084
W20260924 04:51:04.108525 22069 connection.cc:535] client connection to 127.0.98.62:46231 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46231): Connection reset by peer (error 104)
2026-09-24T04:51:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:51:04 dist-test-slave-hvcl krb5kdc[22183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:51:04 dist-test-slave-hvcl krb5kdc[22183](info): set up 2 sockets
Sep 24 04:51:04 dist-test-slave-hvcl krb5kdc[22183](info): commencing operation
krb5kdc: starting...
W20260924 04:51:05.054886 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:51:05 dist-test-slave-hvcl krb5kdc[22183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:51:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:51:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:51:05.173991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33735
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:51:05.286046 22196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:51:05.286332 22196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:51:05.286375 22196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:51:05.290473 22196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:51:05.290549 22196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:51:05.290568 22196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:51:05.290584 22196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:51:05.290598 22196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:51:05.296234 22196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33735
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:51:05.297331 22196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:51:05.298316 22196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:51:05.307013 22196 server_base.cc:1034] running on GCE node
W20260924 04:51:05.306931 22204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:05.306893 22201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:05.306895 22202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:05.307793 22196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:51:05.308917 22196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:51:05.310073 22196 hybrid_clock.cc:648] HybridClock initialized: now 1790225465310054 us; error 31 us; skew 500 ppm
Sep 24 04:51:05 dist-test-slave-hvcl krb5kdc[22183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:51:05.312994 22196 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:51:05.313961 22196 webserver.cc:460] Webserver started at http://127.0.98.62:33225/ using document root <none> and password file <none>
I20260924 04:51:05.314529 22196 fs_manager.cc:359] Metadata directory not provided
I20260924 04:51:05.314597 22196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:51:05.314788 22196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:51:05.316953 22196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "8292f5af3fbc40238c7198d3de89f3c5"
format_stamp: "Formatted at 2026-09-24 04:51:05 on dist-test-slave-hvcl"
server_key: "c080d5b478fba1f97f16582cba17f237"
server_key_iv: "0c327df20b1fad60ddb173a211727fae"
server_key_version: "encryptionkey@0"
I20260924 04:51:05.317530 22196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "8292f5af3fbc40238c7198d3de89f3c5"
format_stamp: "Formatted at 2026-09-24 04:51:05 on dist-test-slave-hvcl"
server_key: "c080d5b478fba1f97f16582cba17f237"
server_key_iv: "0c327df20b1fad60ddb173a211727fae"
server_key_version: "encryptionkey@0"
I20260924 04:51:05.321367 22196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:51:05.323998 22211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:51:05.325263 22196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:51:05.325397 22196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "8292f5af3fbc40238c7198d3de89f3c5"
format_stamp: "Formatted at 2026-09-24 04:51:05 on dist-test-slave-hvcl"
server_key: "c080d5b478fba1f97f16582cba17f237"
server_key_iv: "0c327df20b1fad60ddb173a211727fae"
server_key_version: "encryptionkey@0"
I20260924 04:51:05.325521 22196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:51:05.330093 22196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:51:05.332955 22196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:51:05.333122 22196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:51:05.339668 22196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33735
I20260924 04:51:05.339798 22263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33735 every 8 connection(s)
I20260924 04:51:05.340649 22196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:51:05.343510 22264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:51:05.349131 22264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5: Bootstrap starting.
I20260924 04:51:05.350064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 22196
I20260924 04:51:05.350198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:51:05.350433 392 external_mini_cluster.cc:1428] Setting key eaaaff9e52d18bd3553c7206903dd81d
I20260924 04:51:05.351895 22264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5: Neither blocks nor log segments found. Creating new log.
I20260924 04:51:05.352777 22264 log.cc:824] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:51:05.355028 22264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5: No bootstrap required, opened a new log
I20260924 04:51:05.355777 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:05.351551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51673 (local address 127.0.98.62:33735)
0924 04:51:05.352092 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:05.352122 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:05.352170 (+ 48us) server_negotiation.cc:407] Connection header received
0924 04:51:05.353015 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:05.353039 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:05.353382 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:05.353729 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:05.354366 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:05.354372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:05.354386 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:51:05.354395 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:05.354451 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:51:05.354489 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:05.354591 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:05.354672 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 04:51:05.354827 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":170,"threads_started":1}
I20260924 04:51:05.358559 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:51:05.358853 22264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8292f5af3fbc40238c7198d3de89f3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } }
I20260924 04:51:05.359162 22264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:51:05.359226 22264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8292f5af3fbc40238c7198d3de89f3c5, State: Initialized, Role: FOLLOWER
I20260924 04:51:05.359704 22264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8292f5af3fbc40238c7198d3de89f3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } }
I20260924 04:51:05.359867 22264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:51:05.359938 22264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:51:05.360044 22264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:51:05.361152 22264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8292f5af3fbc40238c7198d3de89f3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } }
I20260924 04:51:05.361515 22264 leader_election.cc:302] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8292f5af3fbc40238c7198d3de89f3c5; no voters:
I20260924 04:51:05.361850 22264 leader_election.cc:288] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:51:05.361943 22274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:51:05.362228 22274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [term 1 LEADER]: Becoming Leader. State: Replica: 8292f5af3fbc40238c7198d3de89f3c5, State: Running, Role: LEADER
I20260924 04:51:05.362546 22274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8292f5af3fbc40238c7198d3de89f3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } }
I20260924 04:51:05.362757 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:05.360913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43067 (local address 127.0.98.62:33735)
0924 04:51:05.361037 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:05.361042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:05.361245 (+ 203us) server_negotiation.cc:407] Connection header received
0924 04:51:05.361364 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:05.361367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:05.361404 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:05.361467 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:05.362317 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:05.362321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:05.362324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:51:05.362336 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:05.362364 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:51:05.362377 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:05.362457 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:05.362607 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 04:51:05.362646 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:51:05.363245 22264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:51:05.364401 22276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8292f5af3fbc40238c7198d3de89f3c5. Latest consensus state: current_term: 1 leader_uuid: "8292f5af3fbc40238c7198d3de89f3c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8292f5af3fbc40238c7198d3de89f3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } } }
I20260924 04:51:05.364429 22275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8292f5af3fbc40238c7198d3de89f3c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8292f5af3fbc40238c7198d3de89f3c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } } }
I20260924 04:51:05.364571 22275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:05.364838 22276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5 [sys.catalog]: This master's current role is: LEADER
W20260924 04:51:05.368162 22289 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:51:05.368233 22289 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:51:05.368381 22284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:51:05.369212 22284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:51:05.375136 22284 catalog_manager.cc:1347] Generated new cluster ID: 1e3d9d678d9d4ecbadcf9744375fc0a0
I20260924 04:51:05.375209 22284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:51:05.398391 22284 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:51:05.398733 22284 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:51:05.404532 22284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8292f5af3fbc40238c7198d3de89f3c5: Generated new TSK 0
I20260924 04:51:05.405278 22284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:51:05.413843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 22196
2026-09-24T04:51:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:51:05 dist-test-slave-hvcl krb5kdc[22295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:51:05 dist-test-slave-hvcl krb5kdc[22295](info): set up 2 sockets
Sep 24 04:51:05 dist-test-slave-hvcl krb5kdc[22295](info): commencing operation
krb5kdc: starting...
W20260924 04:51:06.367100 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:51:06 dist-test-slave-hvcl krb5kdc[22295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:51:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:51:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:51:06.485193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34169
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:51:06.594508 22308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:51:06.594797 22308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:51:06.594841 22308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:51:06.598934 22308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:51:06.599038 22308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:51:06.599058 22308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:51:06.599074 22308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:51:06.599087 22308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:51:06.604447 22308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:51:06.605507 22308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:51:06.606513 22308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:51:06.614825 22313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:06.614818 22314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:06.614894 22316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:06.614910 22308 server_base.cc:1034] running on GCE node
I20260924 04:51:06.615664 22308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:51:06.616838 22308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:51:06.618016 22308 hybrid_clock.cc:648] HybridClock initialized: now 1790225466617988 us; error 40 us; skew 500 ppm
Sep 24 04:51:06 dist-test-slave-hvcl krb5kdc[22295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:51:06.624379 22308 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:51:06.625334 22308 webserver.cc:460] Webserver started at http://127.0.98.62:33947/ using document root <none> and password file <none>
I20260924 04:51:06.625864 22308 fs_manager.cc:359] Metadata directory not provided
I20260924 04:51:06.625921 22308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:51:06.626084 22308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:51:06.628139 22308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "71e5f3df542a4ca1974d9c7872b77e54"
format_stamp: "Formatted at 2026-09-24 04:51:06 on dist-test-slave-hvcl"
server_key: "c55e7dcbfeb537eafeadaf152677fb5f"
server_key_iv: "5cbb18b874d54fc140170625c6e0ad6e"
server_key_version: "encryptionkey@0"
I20260924 04:51:06.628641 22308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "71e5f3df542a4ca1974d9c7872b77e54"
format_stamp: "Formatted at 2026-09-24 04:51:06 on dist-test-slave-hvcl"
server_key: "c55e7dcbfeb537eafeadaf152677fb5f"
server_key_iv: "5cbb18b874d54fc140170625c6e0ad6e"
server_key_version: "encryptionkey@0"
I20260924 04:51:06.632264 22308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:51:06.634621 22323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:51:06.635931 22308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:51:06.636037 22308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "71e5f3df542a4ca1974d9c7872b77e54"
format_stamp: "Formatted at 2026-09-24 04:51:06 on dist-test-slave-hvcl"
server_key: "c55e7dcbfeb537eafeadaf152677fb5f"
server_key_iv: "5cbb18b874d54fc140170625c6e0ad6e"
server_key_version: "encryptionkey@0"
I20260924 04:51:06.636137 22308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:51:06.660089 22308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:51:06.663166 22308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:51:06.663338 22308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:51:06.670055 22308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34169
I20260924 04:51:06.670076 22375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34169 every 8 connection(s)
I20260924 04:51:06.671025 22308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:51:06.673871 22376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:51:06.679287 22376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54: Bootstrap starting.
I20260924 04:51:06.681303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 22308
I20260924 04:51:06.681434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:51:06.681670 392 external_mini_cluster.cc:1428] Setting key ef7457e1d49f1dc0d487853f0c5dd175
I20260924 04:51:06.681722 22376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54: Neither blocks nor log segments found. Creating new log.
I20260924 04:51:06.682534 22376 log.cc:824] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54: Log is configured to *not* fsync() on all Append() calls
I20260924 04:51:06.684448 22376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54: No bootstrap required, opened a new log
I20260924 04:51:06.688470 22376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e5f3df542a4ca1974d9c7872b77e54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34169 } }
I20260924 04:51:06.688747 22376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:51:06.688802 22376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e5f3df542a4ca1974d9c7872b77e54, State: Initialized, Role: FOLLOWER
I20260924 04:51:06.689329 22376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e5f3df542a4ca1974d9c7872b77e54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34169 } }
I20260924 04:51:06.689503 22376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:51:06.689572 22376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:51:06.689670 22376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 04:51:06 dist-test-slave-hvcl krb5kdc[22295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:51:06.690663 22376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e5f3df542a4ca1974d9c7872b77e54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34169 } }
I20260924 04:51:06.691042 22376 leader_election.cc:302] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71e5f3df542a4ca1974d9c7872b77e54; no voters:
I20260924 04:51:06.691430 22376 leader_election.cc:288] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:51:06.691560 22381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:51:06.691776 22381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [term 1 LEADER]: Becoming Leader. State: Replica: 71e5f3df542a4ca1974d9c7872b77e54, State: Running, Role: LEADER
I20260924 04:51:06.692178 22381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e5f3df542a4ca1974d9c7872b77e54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34169 } }
I20260924 04:51:06.692899 22376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:51:06.693874 22382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71e5f3df542a4ca1974d9c7872b77e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e5f3df542a4ca1974d9c7872b77e54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34169 } } }
I20260924 04:51:06.694000 22382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:06.694272 22383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71e5f3df542a4ca1974d9c7872b77e54. Latest consensus state: current_term: 1 leader_uuid: "71e5f3df542a4ca1974d9c7872b77e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e5f3df542a4ca1974d9c7872b77e54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34169 } } }
I20260924 04:51:06.694420 22383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54 [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:06.694753 22390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:51:06.696378 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:06.682727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49127 (local address 127.0.98.62:34169)
0924 04:51:06.683142 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:06.683159 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:06.683196 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:51:06.683908 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:06.683928 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:06.684264 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:06.684553 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:06.685400 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:06.686433 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:06.687660 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:06.688144 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:06.688277 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:06.690940 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:06.690947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:06.690989 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:06.691023 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:06.693058 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:06.693593 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:06.693599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:06.693606 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:06.693764 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:06.694090 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:06.694094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:06.694112 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:06.694618 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:06.694832 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:06.695092 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 04:51:06.695395 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":101,"threads_started":1}
I20260924 04:51:06.698137 22390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:51:06.699620 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:51:06.704668 22390 catalog_manager.cc:1347] Generated new cluster ID: 43d51d4a5fdb464980c3ab720d0bcda8
I20260924 04:51:06.704800 22390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:51:06.710836 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:06.701763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48865 (local address 127.0.98.62:34169)
0924 04:51:06.701873 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:06.701878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:06.702097 (+ 219us) server_negotiation.cc:407] Connection header received
0924 04:51:06.702383 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:06.702386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:06.702441 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:06.702526 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:06.703689 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:06.704534 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:06.705692 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:06.706034 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:06.706126 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:06.707733 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:06.707737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:06.707740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:06.707765 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:06.709295 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:06.709749 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:06.709752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:06.709753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:06.709828 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:06.710061 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:06.710064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:06.710066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:06.710248 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:06.710358 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:06.710564 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 04:51:06.710681 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:51:06.718394 22390 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:51:06.719719 22390 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:51:06.726912 22390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71e5f3df542a4ca1974d9c7872b77e54: Generated new TSK 0
I20260924 04:51:06.727830 22390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:51:06.735288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 22308
2026-09-24T04:51:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:51:06 dist-test-slave-hvcl krb5kdc[22407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:51:06 dist-test-slave-hvcl krb5kdc[22407](info): set up 2 sockets
Sep 24 04:51:06 dist-test-slave-hvcl krb5kdc[22407](info): commencing operation
krb5kdc: starting...
W20260924 04:51:07.703075 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:51:07 dist-test-slave-hvcl krb5kdc[22407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:51:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:51:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:51:07.822052 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44385
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:51:07.931334 22420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:51:07.931653 22420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:51:07.931710 22420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:51:07.935873 22420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:51:07.935945 22420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:51:07.935963 22420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:51:07.935978 22420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:51:07.935993 22420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:51:07.941455 22420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:51:07.942600 22420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:51:07.943663 22420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:51:07.952329 22428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:07.952330 22425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:07.952333 22426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:07.952703 22420 server_base.cc:1034] running on GCE node
I20260924 04:51:07.953274 22420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:51:07.954447 22420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:51:07.955595 22420 hybrid_clock.cc:648] HybridClock initialized: now 1790225467955569 us; error 46 us; skew 500 ppm
Sep 24 04:51:07 dist-test-slave-hvcl krb5kdc[22407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:51:07.958271 22420 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:51:07.959213 22420 webserver.cc:460] Webserver started at http://127.0.98.62:42587/ using document root <none> and password file <none>
I20260924 04:51:07.959749 22420 fs_manager.cc:359] Metadata directory not provided
I20260924 04:51:07.959813 22420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:51:07.960001 22420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:51:07.962067 22420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "c6747c2b66224892854d5ec3838d8441"
format_stamp: "Formatted at 2026-09-24 04:51:07 on dist-test-slave-hvcl"
server_key: "d572a4ff384dcfc79df23006c27c42a9"
server_key_iv: "351f0acb4797b340c2bd281a3b8fb4d2"
server_key_version: "encryptionkey@0"
I20260924 04:51:07.962617 22420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "c6747c2b66224892854d5ec3838d8441"
format_stamp: "Formatted at 2026-09-24 04:51:07 on dist-test-slave-hvcl"
server_key: "d572a4ff384dcfc79df23006c27c42a9"
server_key_iv: "351f0acb4797b340c2bd281a3b8fb4d2"
server_key_version: "encryptionkey@0"
I20260924 04:51:07.966351 22420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:51:07.968909 22435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:51:07.970250 22420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:51:07.970376 22420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "c6747c2b66224892854d5ec3838d8441"
format_stamp: "Formatted at 2026-09-24 04:51:07 on dist-test-slave-hvcl"
server_key: "d572a4ff384dcfc79df23006c27c42a9"
server_key_iv: "351f0acb4797b340c2bd281a3b8fb4d2"
server_key_version: "encryptionkey@0"
I20260924 04:51:07.970482 22420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:51:07.984570 22420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:51:07.987679 22420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:51:07.987866 22420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:51:07.994771 22420 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44385
I20260924 04:51:07.994801 22487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44385 every 8 connection(s)
I20260924 04:51:07.995771 22420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:51:07.998281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 22420
I20260924 04:51:07.998407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:51:07.998610 22488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:51:07.998669 392 external_mini_cluster.cc:1428] Setting key ff588ed51267e5edb7d81a2ce8566883
I20260924 04:51:08.005705 22488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441: Bootstrap starting.
I20260924 04:51:08.008327 22488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441: Neither blocks nor log segments found. Creating new log.
I20260924 04:51:08.008692 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:08.000165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47785 (local address 127.0.98.62:44385)
0924 04:51:08.000599 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:08.000618 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:08.000660 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:51:08.001310 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:08.001334 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:08.001772 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:08.002139 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:08.002566 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:08.004014 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:08.004932 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:08.005615 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:08.005782 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:08.006602 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:08.006607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:08.006622 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:51:08.007175 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:08.007244 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:51:08.007283 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:08.007375 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:08.007526 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 04:51:08.007733 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":118,"threads_started":1}
I20260924 04:51:08.009159 22488 log.cc:824] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441: Log is configured to *not* fsync() on all Append() calls
I20260924 04:51:08.011101 22488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441: No bootstrap required, opened a new log
I20260924 04:51:08.011494 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:51:08.015220 22488 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6747c2b66224892854d5ec3838d8441" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44385 } }
I20260924 04:51:08.015465 22488 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:51:08.015532 22488 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6747c2b66224892854d5ec3838d8441, State: Initialized, Role: FOLLOWER
I20260924 04:51:08.016033 22488 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6747c2b66224892854d5ec3838d8441" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44385 } }
I20260924 04:51:08.016203 22488 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:51:08.016269 22488 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:51:08.016368 22488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:51:08.017401 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:08.013379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38155 (local address 127.0.98.62:44385)
0924 04:51:08.013495 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:08.013499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:08.013574 (+ 75us) server_negotiation.cc:407] Connection header received
0924 04:51:08.013675 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:08.013678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:08.013725 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:08.013810 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:08.014017 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:08.014797 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:08.015631 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:08.015960 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:08.016061 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:08.016946 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:08.016949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:08.016952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:51:08.016965 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:08.016992 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:51:08.017004 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:08.017077 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:08.017204 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 04:51:08.017262 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:51:08.017450 22488 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6747c2b66224892854d5ec3838d8441" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44385 } }
I20260924 04:51:08.017784 22488 leader_election.cc:302] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6747c2b66224892854d5ec3838d8441; no voters:
I20260924 04:51:08.018091 22488 leader_election.cc:288] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:51:08.018173 22498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:51:08.018383 22498 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [term 1 LEADER]: Becoming Leader. State: Replica: c6747c2b66224892854d5ec3838d8441, State: Running, Role: LEADER
I20260924 04:51:08.018744 22498 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6747c2b66224892854d5ec3838d8441" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44385 } }
I20260924 04:51:08.019495 22488 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:51:08.019995 22498 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6747c2b66224892854d5ec3838d8441. Latest consensus state: current_term: 1 leader_uuid: "c6747c2b66224892854d5ec3838d8441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6747c2b66224892854d5ec3838d8441" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44385 } } }
I20260924 04:51:08.020203 22498 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:08.020385 22499 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6747c2b66224892854d5ec3838d8441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6747c2b66224892854d5ec3838d8441" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44385 } } }
I20260924 04:51:08.020491 22499 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441 [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:08.020625 22505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:51:08.023638 22505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:51:08.028612 22505 catalog_manager.cc:1347] Generated new cluster ID: 338c481960cd4399bf3d5ef7512e2009
I20260924 04:51:08.028683 22505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:51:08.050448 22505 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:51:08.051822 22505 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:51:08.063582 22505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6747c2b66224892854d5ec3838d8441: Generated new TSK 0
I20260924 04:51:08.064469 22505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:51:08.071069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 22420
2026-09-24T04:51:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:51:08 dist-test-slave-hvcl krb5kdc[22519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:51:08 dist-test-slave-hvcl krb5kdc[22519](info): set up 2 sockets
Sep 24 04:51:08 dist-test-slave-hvcl krb5kdc[22519](info): commencing operation
krb5kdc: starting...
W20260924 04:51:09.015044 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:51:09 dist-test-slave-hvcl krb5kdc[22519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:51:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:51:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:51:09.130437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44373
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:51:09.236864 22532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:51:09.237141 22532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:51:09.237183 22532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:51:09.241288 22532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:51:09.241365 22532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:51:09.241384 22532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:51:09.241400 22532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:51:09.241413 22532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:51:09.246797 22532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:51:09.247874 22532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:51:09.248878 22532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:51:09.257124 22538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:09.257201 22540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:09.257280 22532 server_base.cc:1034] running on GCE node
W20260924 04:51:09.257198 22537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:09.257961 22532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:51:09.259156 22532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:51:09.260324 22532 hybrid_clock.cc:648] HybridClock initialized: now 1790225469260306 us; error 31 us; skew 500 ppm
Sep 24 04:51:09 dist-test-slave-hvcl krb5kdc[22519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:51:09.263060 22532 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:51:09.263986 22532 webserver.cc:460] Webserver started at http://127.0.98.62:38571/ using document root <none> and password file <none>
I20260924 04:51:09.264545 22532 fs_manager.cc:359] Metadata directory not provided
I20260924 04:51:09.264611 22532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:51:09.264801 22532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:51:09.266850 22532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "d4c4e947920d4fd3af98faee502b590e"
format_stamp: "Formatted at 2026-09-24 04:51:09 on dist-test-slave-hvcl"
server_key: "ebd46f48ea36ee31ca5cd6912911ccca"
server_key_iv: "58906bc227adcd89360e6b592396af20"
server_key_version: "encryptionkey@0"
I20260924 04:51:09.267417 22532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4c4e947920d4fd3af98faee502b590e"
format_stamp: "Formatted at 2026-09-24 04:51:09 on dist-test-slave-hvcl"
server_key: "ebd46f48ea36ee31ca5cd6912911ccca"
server_key_iv: "58906bc227adcd89360e6b592396af20"
server_key_version: "encryptionkey@0"
I20260924 04:51:09.271090 22532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:51:09.273554 22547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:51:09.274842 22532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:51:09.275013 22532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "d4c4e947920d4fd3af98faee502b590e"
format_stamp: "Formatted at 2026-09-24 04:51:09 on dist-test-slave-hvcl"
server_key: "ebd46f48ea36ee31ca5cd6912911ccca"
server_key_iv: "58906bc227adcd89360e6b592396af20"
server_key_version: "encryptionkey@0"
I20260924 04:51:09.275141 22532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:51:09.280730 22532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:51:09.283840 22532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:51:09.283996 22532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:51:09.290561 22532 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44373
I20260924 04:51:09.290594 22599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44373 every 8 connection(s)
I20260924 04:51:09.291520 22532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:51:09.294288 22600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:51:09.296355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 22532
I20260924 04:51:09.296496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:51:09.296774 392 external_mini_cluster.cc:1428] Setting key c1fe4562c01cc41be076fcbb033be6e0
I20260924 04:51:09.301127 22600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e: Bootstrap starting.
I20260924 04:51:09.302083 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:09.298224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42235 (local address 127.0.98.62:44373)
0924 04:51:09.298700 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:09.298716 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:09.298753 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:51:09.299454 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:09.299480 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:09.299787 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:09.300085 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:09.300890 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:09.300894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:09.300907 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:51:09.300917 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:09.300981 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:51:09.301001 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:09.301123 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:09.301214 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 04:51:09.301369 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":141,"threads_started":1}
I20260924 04:51:09.303445 22600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e: Neither blocks nor log segments found. Creating new log.
I20260924 04:51:09.304198 22600 log.cc:824] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:51:09.304682 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:51:09.306067 22600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e: No bootstrap required, opened a new log
I20260924 04:51:09.308118 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:09.306548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52781 (local address 127.0.98.62:44373)
0924 04:51:09.306647 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:09.306651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:09.306869 (+ 218us) server_negotiation.cc:407] Connection header received
0924 04:51:09.307029 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:09.307031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:09.307061 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:09.307120 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:09.307773 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:09.307776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:09.307779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:51:09.307790 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:09.307817 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:51:09.307827 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:09.307895 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:09.307967 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 04:51:09.308005 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 04:51:09.309871 22600 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c4e947920d4fd3af98faee502b590e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44373 } }
I20260924 04:51:09.310101 22600 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:51:09.310153 22600 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4c4e947920d4fd3af98faee502b590e, State: Initialized, Role: FOLLOWER
I20260924 04:51:09.310564 22600 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c4e947920d4fd3af98faee502b590e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44373 } }
I20260924 04:51:09.310691 22600 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:51:09.310752 22600 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:51:09.310846 22600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:51:09.311815 22600 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c4e947920d4fd3af98faee502b590e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44373 } }
I20260924 04:51:09.312155 22600 leader_election.cc:302] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4c4e947920d4fd3af98faee502b590e; no voters:
I20260924 04:51:09.312479 22600 leader_election.cc:288] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:51:09.312577 22610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:51:09.312827 22610 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [term 1 LEADER]: Becoming Leader. State: Replica: d4c4e947920d4fd3af98faee502b590e, State: Running, Role: LEADER
I20260924 04:51:09.313172 22610 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c4e947920d4fd3af98faee502b590e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44373 } }
I20260924 04:51:09.313743 22600 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:51:09.314746 22612 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4c4e947920d4fd3af98faee502b590e. Latest consensus state: current_term: 1 leader_uuid: "d4c4e947920d4fd3af98faee502b590e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c4e947920d4fd3af98faee502b590e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44373 } } }
I20260924 04:51:09.314874 22612 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:09.315165 22611 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4c4e947920d4fd3af98faee502b590e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4c4e947920d4fd3af98faee502b590e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44373 } } }
I20260924 04:51:09.315243 22619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:51:09.315250 22611 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:09.318061 22619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:51:09.323514 22619 catalog_manager.cc:1347] Generated new cluster ID: 6f3f74d0679344eeb58a763003734835
I20260924 04:51:09.323606 22619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:51:09.384087 22619 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:51:09.384470 22619 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:51:09.395141 22619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4c4e947920d4fd3af98faee502b590e: Generated new TSK 0
I20260924 04:51:09.396019 22619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:51:09.405558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 22532
2026-09-24T04:51:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1334 ms)
[----------] 18 tests from AuthTokenIssuingTest (23905 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:51:09 dist-test-slave-hvcl krb5kdc[22631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:51:09 dist-test-slave-hvcl krb5kdc[22631](info): set up 2 sockets
Sep 24 04:51:09 dist-test-slave-hvcl krb5kdc[22631](info): commencing operation
krb5kdc: starting...
W20260924 04:51:10.367178 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:51:10 dist-test-slave-hvcl krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:51:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:51:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:51:10.483923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36993
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36993,127.0.98.61:43123,127.0.98.60:45223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:51:10.596210 22644 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:51:10.596516 22644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:51:10.596539 22644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:51:10.596585 22644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:51:10.600967 22644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:51:10.601043 22644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:51:10.601063 22644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:51:10.601080 22644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:51:10.601094 22644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:51:10.606690 22644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36993,127.0.98.61:43123,127.0.98.60:45223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:51:10.607831 22644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:51:10.608815 22644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:51:10.617555 22652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:10.617568 22650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:10.617686 22644 server_base.cc:1034] running on GCE node
W20260924 04:51:10.617573 22649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:10.618530 22644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:51:10.619699 22644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:51:10.620905 22644 hybrid_clock.cc:648] HybridClock initialized: now 1790225470620889 us; error 31 us; skew 500 ppm
Sep 24 04:51:10 dist-test-slave-hvcl krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:51:10.623731 22644 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:51:10.624742 22644 webserver.cc:460] Webserver started at http://127.0.98.62:37493/ using document root <none> and password file <none>
I20260924 04:51:10.625321 22644 fs_manager.cc:359] Metadata directory not provided
I20260924 04:51:10.625392 22644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:51:10.625587 22644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:51:10.627804 22644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "6c3815267d324788a0ab1dc0b2d55d97"
format_stamp: "Formatted at 2026-09-24 04:51:10 on dist-test-slave-hvcl"
server_key: "1cbaf0d4b486d294148cb7de0c0232b9"
server_key_iv: "fd057679e33d91978558e94a00d370d0"
server_key_version: "encryptionkey@0"
I20260924 04:51:10.628381 22644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c3815267d324788a0ab1dc0b2d55d97"
format_stamp: "Formatted at 2026-09-24 04:51:10 on dist-test-slave-hvcl"
server_key: "1cbaf0d4b486d294148cb7de0c0232b9"
server_key_iv: "fd057679e33d91978558e94a00d370d0"
server_key_version: "encryptionkey@0"
I20260924 04:51:10.632310 22644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:51:10.635001 22659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:51:10.636340 22644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:51:10.636472 22644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "6c3815267d324788a0ab1dc0b2d55d97"
format_stamp: "Formatted at 2026-09-24 04:51:10 on dist-test-slave-hvcl"
server_key: "1cbaf0d4b486d294148cb7de0c0232b9"
server_key_iv: "fd057679e33d91978558e94a00d370d0"
server_key_version: "encryptionkey@0"
I20260924 04:51:10.636583 22644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:51:10.647411 22644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:51:10.650557 22644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:51:10.650738 22644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:51:10.657380 22644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36993
I20260924 04:51:10.657635 22711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36993 every 8 connection(s)
I20260924 04:51:10.658349 22644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:51:10.660166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 22644
I20260924 04:51:10.660305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:51:10.660591 392 external_mini_cluster.cc:1428] Setting key 3690dafe9eacf8be3ea69df426281893
I20260924 04:51:10.661465 22712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:51:10.666241 22712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 04:51:10 dist-test-slave-hvcl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:51:10.682061 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.669143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.0.98.62:36993)
0924 04:51:10.669672 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:10.669702 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:10.669815 (+ 113us) server_negotiation.cc:407] Connection header received
0924 04:51:10.670924 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:10.670929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:10.671193 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:10.671359 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:10.672241 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.673215 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.674256 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.674633 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.674785 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:10.677104 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:10.677110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:10.677123 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:10.677144 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:10.678736 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:10.679414 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:10.679419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:10.679421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:10.679543 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:10.679921 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:10.679923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:10.679924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:10.680362 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:10.680482 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:10.680879 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 04:51:10.681116 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":387,"spinlock_wait_cycles":62336,"thread_start_us":149,"threads_started":1}
I20260924 04:51:10.682066 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.669143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36993 (local address 127.0.0.1:49198)
0924 04:51:10.669676 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.669716 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:10.670615 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:10.671369 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:10.671374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:10.671716 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:10.672089 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:10.672108 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.673346 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.673349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:10.674126 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:10.674148 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.674783 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.674787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:10.674893 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:10.675486 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:10.675508 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:10.676964 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:10.678917 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:10.678928 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:10.678940 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:10.679253 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:10.679716 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:10.679719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:10.679720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:10.679840 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:10.680478 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:10.680484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:10.680628 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 04:51:10.680794 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 04:51:10.681217 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":122,"thread_start_us":134,"threads_started":1}
I20260924 04:51:10.684548 22712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:51:10.685321 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.685054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:59794)
0924 04:51:10.685172 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.685256 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 04:51:10.685526 22663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
I20260924 04:51:10.686574 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.686306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:59802)
0924 04:51:10.686416 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.686466 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:51:10.686770 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43123: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:51:10.688167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43123
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36993,127.0.98.61:43123,127.0.98.60:45223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:51:10.736016 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } attempt: 1
I20260924 04:51:10.736954 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.736634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:59814)
0924 04:51:10.736808 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.736878 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 04:51:10.737617 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.737418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:59824)
0924 04:51:10.737516 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.737560 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 04:51:10.737731 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43123: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
W20260924 04:51:10.798379 22719 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:51:10.798700 22719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:51:10.798735 22719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:51:10.798801 22719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:51:10.802948 22719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:51:10.803054 22719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:51:10.803081 22719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:51:10.803102 22719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:51:10.803122 22719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:51:10.805897 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } attempt: 2
I20260924 04:51:10.806792 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.806427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:59830)
0924 04:51:10.806595 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.806661 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 04:51:10.807497 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.807295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:59840)
0924 04:51:10.807396 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.807442 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 04:51:10.807607 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43123: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
I20260924 04:51:10.808954 22719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36993,127.0.98.61:43123,127.0.98.60:45223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:51:10.810052 22719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:51:10.811112 22719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:51:10.819860 22725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:10.819947 22719 server_base.cc:1034] running on GCE node
W20260924 04:51:10.819855 22724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:10.819862 22727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:10.820667 22719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:51:10.821820 22719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:51:10.823021 22719 hybrid_clock.cc:648] HybridClock initialized: now 1790225470822988 us; error 43 us; skew 500 ppm
Sep 24 04:51:10 dist-test-slave-hvcl krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:51:10.825995 22719 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:51:10.827025 22719 webserver.cc:460] Webserver started at http://127.0.98.61:32955/ using document root <none> and password file <none>
I20260924 04:51:10.827591 22719 fs_manager.cc:359] Metadata directory not provided
I20260924 04:51:10.827661 22719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:51:10.827963 22719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:51:10.830054 22719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/data/instance:
uuid: "3bc38459cf5847fda080bdbb9df217f9"
format_stamp: "Formatted at 2026-09-24 04:51:10 on dist-test-slave-hvcl"
server_key: "3eb29a71b235d8a8caeaa5f438d49335"
server_key_iv: "19c8a945aedc0c6789e04d4fd3e713a8"
server_key_version: "encryptionkey@0"
I20260924 04:51:10.830607 22719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/wal/instance:
uuid: "3bc38459cf5847fda080bdbb9df217f9"
format_stamp: "Formatted at 2026-09-24 04:51:10 on dist-test-slave-hvcl"
server_key: "3eb29a71b235d8a8caeaa5f438d49335"
server_key_iv: "19c8a945aedc0c6789e04d4fd3e713a8"
server_key_version: "encryptionkey@0"
I20260924 04:51:10.834435 22719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:51:10.837025 22734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:51:10.838317 22719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:51:10.838438 22719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/wal
uuid: "3bc38459cf5847fda080bdbb9df217f9"
format_stamp: "Formatted at 2026-09-24 04:51:10 on dist-test-slave-hvcl"
server_key: "3eb29a71b235d8a8caeaa5f438d49335"
server_key_iv: "19c8a945aedc0c6789e04d4fd3e713a8"
server_key_version: "encryptionkey@0"
I20260924 04:51:10.838541 22719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:51:10.857631 22719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:51:10.860803 22719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:51:10.861001 22719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:51:10.867828 22719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43123
I20260924 04:51:10.868086 22786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43123 every 8 connection(s)
I20260924 04:51:10.868830 22719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:51:10.871698 22787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:51:10.874714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 22719
I20260924 04:51:10.874868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-1/wal/instance
I20260924 04:51:10.875182 392 external_mini_cluster.cc:1428] Setting key 1498b05b981ff282e0c08fde12feb91f
I20260924 04:51:10.875928 22787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 04:51:10 dist-test-slave-hvcl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:51:10.890359 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.878179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.0.98.62:36993)
0924 04:51:10.878312 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:10.878315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:10.879441 (+ 1126us) server_negotiation.cc:407] Connection header received
0924 04:51:10.880490 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:10.880493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:10.880553 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:10.880625 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:10.881814 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.882357 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.883569 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.883859 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.883954 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:10.886496 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:10.886499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:10.886501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:10.886529 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:10.887990 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:10.888492 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:10.888496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:10.888497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:10.888582 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:10.889041 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:10.889045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:10.889047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:10.889299 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:10.889399 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:10.890046 (+ 647us) server_negotiation.cc:299] Negotiation successful
0924 04:51:10.890183 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:51:10.891189 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.878696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36993 (local address 127.0.0.1:49214)
0924 04:51:10.879254 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.879314 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:10.880266 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:10.880819 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:10.880827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:10.881178 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:10.881663 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:10.881677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.882480 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.882483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:10.883448 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:10.883458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.884010 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.884014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:10.884183 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:10.884760 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:10.884785 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:10.886303 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:10.888099 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:10.888104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:10.888117 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:10.888403 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:10.888762 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:10.888764 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:10.888766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:10.888923 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:10.889412 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:10.889418 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:10.889773 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 04:51:10.889949 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 04:51:10.890246 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":176,"threads_started":1}
I20260924 04:51:10.892825 22787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:51:10 dist-test-slave-hvcl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:51:10.898744 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:51:10.901419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45223
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36993,127.0.98.61:43123,127.0.98.60:45223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:51:10.902338 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.893325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:59846)
0924 04:51:10.893419 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.893433 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:10.893516 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:10.894232 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:10.894234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:10.894363 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:10.894450 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:10.894455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.895669 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.895672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:10.896221 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:10.896230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.896810 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.896813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:10.896883 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:10.897309 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:10.897320 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:10.898453 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:10.900696 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:10.900699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:10.900701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:10.900910 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:10.901191 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:10.901195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:10.901197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:10.901307 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:10.901823 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:10.901826 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:10.901917 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 04:51:10.902019 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 04:51:10.902143 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 04:51:10.902544 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.893461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.0.98.61:43123)
0924 04:51:10.893833 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:10.893842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:10.893890 (+ 48us) server_negotiation.cc:407] Connection header received
0924 04:51:10.893933 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:10.893937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:10.894098 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:10.894208 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:10.894548 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.895575 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.896312 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.896715 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.896799 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:10.898577 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:10.898585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:10.898597 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:10.898633 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:10.900606 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:10.901013 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:10.901020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:10.901021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:10.901109 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:10.901398 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:10.901401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:10.901403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:10.901715 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:10.901843 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:10.902222 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 04:51:10.902378 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":159,"threads_started":1}
I20260924 04:51:10.905017 22787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:51:10 dist-test-slave-hvcl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:51:10.905797 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.905460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40770)
0924 04:51:10.905623 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.905702 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260924 04:51:10.905994 22736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
I20260924 04:51:10.907215 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.906929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40786)
0924 04:51:10.907103 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.907149 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260924 04:51:10.907349 22787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45223: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
I20260924 04:51:10.909199 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.899216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:59852)
0924 04:51:10.899389 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.899405 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:10.899489 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:10.899748 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:10.899751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:10.899893 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:10.900000 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:10.900005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.900955 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.900959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:10.902187 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:10.902199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.903440 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.903444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:10.903553 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:10.904709 (+ 1156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:10.904723 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:10.906050 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:10.907844 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:10.907848 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:10.907850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:10.908039 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:10.908283 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:10.908286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:10.908288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:10.908368 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:10.908709 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:10.908713 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:10.908816 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 04:51:10.908898 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:51:10.909040 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 04:51:10.909220 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.899272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.0.98.61:43123)
0924 04:51:10.899550 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:10.899556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:10.899569 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:51:10.899602 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:10.899605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:10.899644 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:10.899727 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:10.900100 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.900840 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.902310 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:10.903297 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:10.903392 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:10.906208 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:10.906211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:10.906213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:10.906233 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:10.907744 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:10.908125 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:10.908128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:10.908130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:10.908205 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:10.908445 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:10.908448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:10.908450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:10.908625 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:10.908706 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:10.908957 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 04:51:10.909056 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":100,"threads_started":1}
I20260924 04:51:10.909817 22712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:51:10.910470 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.910215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40798)
0924 04:51:10.910341 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.910397 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 04:51:10.911267 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.911001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40804)
0924 04:51:10.911150 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.911201 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260924 04:51:10.911406 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45223: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
I20260924 04:51:10.942564 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } attempt: 1
I20260924 04:51:10.943532 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.943197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40814)
0924 04:51:10.943383 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.943448 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 04:51:10.944231 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.944037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40824)
0924 04:51:10.944126 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.944172 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 04:51:10.944427 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45223: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
I20260924 04:51:10.956632 22787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } attempt: 1
I20260924 04:51:10.957566 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.957236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40838)
0924 04:51:10.957395 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.957470 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 04:51:10.958329 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:10.958106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40840)
0924 04:51:10.958210 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:10.958264 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 04:51:10.958480 22787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45223: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
W20260924 04:51:11.016330 22794 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:51:11.016630 22794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:51:11.016654 22794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:51:11.016701 22794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:51:11.019573 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } attempt: 2
I20260924 04:51:11.020391 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.020099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40844)
0924 04:51:11.020249 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:11.020313 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 04:51:11.021019 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.020831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40860)
0924 04:51:11.020918 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:11.020962 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 04:51:11.021133 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45223: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
W20260924 04:51:11.021468 22794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:51:11.021526 22794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:51:11.021545 22794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:51:11.021567 22794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:51:11.021581 22794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:51:11.026643 22787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } attempt: 2
I20260924 04:51:11.027273 22794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36993,127.0.98.61:43123,127.0.98.60:45223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:51:11.027454 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.027158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40874)
0924 04:51:11.027306 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:11.027374 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 04:51:11.028110 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.027909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40882)
0924 04:51:11.028005 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:11.028052 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 04:51:11.028239 22787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45223: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45223: connect: Connection refused (error 111)
I20260924 04:51:11.028584 22794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:51:11.029589 22794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:51:11.037892 22800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:11.038038 22794 server_base.cc:1034] running on GCE node
W20260924 04:51:11.037892 22801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:11.038242 22803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:11.038842 22794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:51:11.040033 22794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:51:11.041224 22794 hybrid_clock.cc:648] HybridClock initialized: now 1790225471041202 us; error 37 us; skew 500 ppm
Sep 24 04:51:11 dist-test-slave-hvcl krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:51:11.044114 22794 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:51:11.044993 22794 webserver.cc:460] Webserver started at http://127.0.98.60:36515/ using document root <none> and password file <none>
I20260924 04:51:11.045516 22794 fs_manager.cc:359] Metadata directory not provided
I20260924 04:51:11.045578 22794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:51:11.045830 22794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:51:11.047942 22794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/data/instance:
uuid: "f7fb5dcea52647b6b50f568dec2e27c9"
format_stamp: "Formatted at 2026-09-24 04:51:11 on dist-test-slave-hvcl"
server_key: "570cc4bbda44e73a0f6aa241fef42cc7"
server_key_iv: "8c6256442de2d92db00003b6d0894d1e"
server_key_version: "encryptionkey@0"
I20260924 04:51:11.048480 22794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/wal/instance:
uuid: "f7fb5dcea52647b6b50f568dec2e27c9"
format_stamp: "Formatted at 2026-09-24 04:51:11 on dist-test-slave-hvcl"
server_key: "570cc4bbda44e73a0f6aa241fef42cc7"
server_key_iv: "8c6256442de2d92db00003b6d0894d1e"
server_key_version: "encryptionkey@0"
I20260924 04:51:11.052213 22794 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 04:51:11.054661 22810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:51:11.055955 22794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:51:11.056087 22794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/wal
uuid: "f7fb5dcea52647b6b50f568dec2e27c9"
format_stamp: "Formatted at 2026-09-24 04:51:11 on dist-test-slave-hvcl"
server_key: "570cc4bbda44e73a0f6aa241fef42cc7"
server_key_iv: "8c6256442de2d92db00003b6d0894d1e"
server_key_version: "encryptionkey@0"
I20260924 04:51:11.056188 22794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:51:11.068923 22794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:51:11.072278 22794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:51:11.072463 22794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:51:11.079097 22794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45223
I20260924 04:51:11.079355 22862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45223 every 8 connection(s)
I20260924 04:51:11.080144 22794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:51:11.083002 22863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:51:11.086778 22863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:51:11.088438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 22794
I20260924 04:51:11.088537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225194677594-392-0/minicluster-data/master-2/wal/instance
I20260924 04:51:11.088820 392 external_mini_cluster.cc:1428] Setting key 7d26ee91f06ecd102540886bd4de06ed
Sep 24 04:51:11 dist-test-slave-hvcl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 04:51:11 dist-test-slave-hvcl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:51:11.100199 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.088721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.0.98.62:36993)
0924 04:51:11.088864 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.088870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.089786 (+ 916us) server_negotiation.cc:407] Connection header received
0924 04:51:11.090893 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.090896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.090953 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.091063 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.092040 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.092552 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.093595 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.093822 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.093901 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.096487 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:11.096491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:11.096494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:11.096525 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:11.098014 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.098602 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.098605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.098606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.098678 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.099059 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.099062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.099064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.099278 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:11.099376 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:11.099908 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 04:51:11.100032 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:51:11.101197 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.089082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36993 (local address 127.0.0.1:49218)
0924 04:51:11.089624 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:11.089684 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:11.090662 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:11.091135 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:11.091140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:11.091614 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:11.091901 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:11.091913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.092653 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.092656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:11.093477 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:11.093486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.093959 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.093962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:11.094133 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.094755 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:11.094770 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:11.096320 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:11.098115 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:11.098122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:11.098133 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:11.098498 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:11.098823 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:11.098825 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:11.098826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:11.098956 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:11.099375 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:11.099380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:11.099689 (+ 309us) client_negotiation.cc:769] Sending connection context
0924 04:51:11.099858 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 04:51:11.100183 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":18,"thread_start_us":155,"threads_started":1}
I20260924 04:51:11.101202 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.089453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.0.98.60:45223)
0924 04:51:11.089919 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.089929 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.089966 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:51:11.090651 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.090677 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.090923 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.091092 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.091857 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.092789 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.093605 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.094008 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.094128 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.096931 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:11.096938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:11.096952 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:11.096977 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:11.098672 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.099081 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.099087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.099089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.099157 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.099426 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.099429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.099430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.099696 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:11.099922 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:11.100054 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 04:51:11.100574 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":385,"thread_start_us":129,"threads_started":1}
I20260924 04:51:11.102455 22863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:51:11 dist-test-slave-hvcl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:51:11 dist-test-slave-hvcl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:51:11.111155 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.102964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:59860)
0924 04:51:11.103087 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:11.103101 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:11.103171 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:11.103394 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:11.103397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:11.103554 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:11.103665 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:11.103673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.104706 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.104709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:11.105280 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:11.105290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.105717 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.105720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:11.105817 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.106237 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:11.106251 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:11.107668 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:11.109723 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:11.109756 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:11.109757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:11.109940 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:11.110199 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:11.110201 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:11.110202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:11.110280 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:11.110634 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:11.110636 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:11.110717 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 04:51:11.110882 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 04:51:11.111034 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 04:51:11.111454 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.103085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59860 (local address 127.0.98.61:43123)
0924 04:51:11.103199 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.103204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.103218 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:51:11.103257 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.103259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.103304 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.103369 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.103804 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.104580 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.105385 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.105623 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.105700 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.107853 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:11.107857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:11.107860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:11.107890 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:11.109563 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.110033 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.110036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.110038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.110122 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.110364 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.110367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.110368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.110543 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:11.110645 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:11.111003 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 04:51:11.111139 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:51:11.112102 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.103683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.0.98.62:36993)
0924 04:51:11.103779 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.103782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.103796 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:51:11.103900 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.103902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.103936 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.103988 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.104627 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.105080 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.105910 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.106107 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.106190 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.108764 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:11.108768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:11.108771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:11.108797 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:11.110216 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.110588 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.110592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.110594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.110671 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.110944 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.110948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.110950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.111194 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:11.111425 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:11.111750 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 04:51:11.111843 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 04:51:11.112238 22863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:51:11 dist-test-slave-hvcl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:51:11 dist-test-slave-hvcl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:51:11.119400 22787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } attempt: 3
I20260924 04:51:11.119681 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.112740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40894)
0924 04:51:11.112835 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:11.112847 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:11.112908 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:11.113389 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:11.113392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:11.113529 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:11.113588 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:11.113592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.114252 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.114255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:11.114744 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:11.114751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.115211 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.115214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:11.115285 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.115680 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:11.115693 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:11.116799 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:11.118229 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:11.118232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:11.118234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:11.118438 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:11.118679 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:11.118682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:11.118684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:11.118762 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:11.119197 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:11.119200 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:11.119295 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 04:51:11.119403 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 04:51:11.119518 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 04:51:11.119936 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.112864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40894 (local address 127.0.98.60:45223)
0924 04:51:11.112964 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.112967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.112978 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:51:11.113268 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.113271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.113313 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.113370 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.113677 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.114165 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.114836 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.115115 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.115193 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.116929 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:11.116933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:11.116936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:11.116960 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:11.118128 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.118532 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.118535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.118536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.118607 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.118848 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.118851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.118853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.119107 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:11.119184 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:11.119696 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 04:51:11.119805 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:51:11.120280 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } attempt: 3
I20260924 04:51:11.123704 22863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9: Bootstrap starting.
Sep 24 04:51:11 dist-test-slave-hvcl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:51:11 dist-test-slave-hvcl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:51:11.126600 22863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9: Neither blocks nor log segments found. Creating new log.
I20260924 04:51:11.127532 22863 log.cc:824] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:51:11.127897 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.119787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40902)
0924 04:51:11.119881 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:11.119894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:11.119961 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:11.120236 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:11.120239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:11.120399 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:11.120499 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:11.120505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.121375 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.121379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:11.122093 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:11.122104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.122669 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.122673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:11.122749 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.123221 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:11.123235 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:11.124530 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:11.126355 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:11.126359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:11.126361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:11.126599 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:11.126889 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:11.126892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:11.126894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:11.127014 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:11.127411 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:11.127414 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:11.127509 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 04:51:11.127591 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:51:11.127743 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 04:51:11.128399 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.119878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.0.98.60:45223)
0924 04:51:11.120060 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.120063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.120075 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:51:11.120106 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.120109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.120155 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.120215 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.120628 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.121261 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.122231 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.122519 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.122947 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.124647 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:11.124651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:11.124654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:11.124676 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:11.126224 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.126701 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.126705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.126706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.126788 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.127094 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.127097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.127098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.127296 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:11.128074 (+ 778us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:11.128147 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 04:51:11.128249 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":146,"threads_started":1}
I20260924 04:51:11.128468 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.120638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45223 (local address 127.0.0.1:40912)
0924 04:51:11.120752 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:11.120765 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:11.120835 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:11.121202 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:11.121205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:11.121353 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:11.121430 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:11.121435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.122442 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.122445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:11.122997 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:11.123007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.123529 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.123533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:11.123611 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.124052 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:11.124066 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:11.125309 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:11.127155 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:11.127158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:11.127160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:11.127337 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:11.127574 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:11.127577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:11.127578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:11.127658 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:11.128004 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:11.128007 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:11.128099 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 04:51:11.128181 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:51:11.128316 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 04:51:11.129058 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.120948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.0.98.60:45223)
0924 04:51:11.121024 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.121032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.121042 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:51:11.121064 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.121066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.121094 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.121165 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.121524 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.122342 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.123106 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.123403 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.123813 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.125442 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:11.125445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:11.125448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:11.125471 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:11.127076 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.127419 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.127422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.127424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.127496 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.127752 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.127754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.127755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.127899 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:11.128739 (+ 840us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:11.128808 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 04:51:11.128911 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 04:51:11.130641 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.113178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.0.98.61:43123)
0924 04:51:11.113283 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.113286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.113298 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:51:11.113372 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.113375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.113411 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.113473 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.114116 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.114610 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.115368 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.115594 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.115678 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.118626 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:11.118630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:11.118633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:11.118657 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:11.128605 (+ 9948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.129075 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.129079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.129081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.129350 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.129817 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.129820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.129822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.130015 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:11.130205 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:11.130402 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 04:51:11.130510 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:51:11.130947 22863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9: No bootstrap required, opened a new log
I20260924 04:51:11.131582 22787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9: Bootstrap starting.
I20260924 04:51:11.133821 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:51:11.134255 22787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9: Neither blocks nor log segments found. Creating new log.
I20260924 04:51:11.134182 22863 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } }
I20260924 04:51:11.134377 22863 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7fb5dcea52647b6b50f568dec2e27c9, State: Initialized, Role: FOLLOWER
I20260924 04:51:11.134683 22863 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } }
I20260924 04:51:11.135114 22870 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } } }
I20260924 04:51:11.135154 22787 log.cc:824] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:51:11.135247 22870 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:51:11.135967 22863 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:51:11.135274 22686 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6c3815267d324788a0ab1dc0b2d55d97"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49226
I20260924 04:51:11.137252 22787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9: No bootstrap required, opened a new log
I20260924 04:51:11.137838 22712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97: Bootstrap starting.
I20260924 04:51:11.140413 22712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97: Neither blocks nor log segments found. Creating new log.
I20260924 04:51:11.141278 22712 log.cc:824] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97: Log is configured to *not* fsync() on all Append() calls
W20260924 04:51:11.141542 22882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:51:11.141662 22882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:51:11.139469 22794 master_runner.cc:186] Error getting master registration for 127.0.98.62:36993: OK, instance_id { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" instance_seqno: 1790225470655542 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:51:11.143244 22794 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:51:11.143301 22712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97: No bootstrap required, opened a new log
I20260924 04:51:11.145551 22787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } }
I20260924 04:51:11.145792 22787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc38459cf5847fda080bdbb9df217f9, State: Initialized, Role: FOLLOWER
I20260924 04:51:11.146140 22787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } }
I20260924 04:51:11.146095 22712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } }
I20260924 04:51:11.146304 22712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3815267d324788a0ab1dc0b2d55d97, State: Initialized, Role: FOLLOWER
I20260924 04:51:11.146598 22712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } }
I20260924 04:51:11.146706 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } } }
I20260924 04:51:11.147089 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:51:11.147544 22787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:51:11.147485 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } } }
I20260924 04:51:11.147766 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:51:11.147871 22712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:51:11.147915 22686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 04:51:11.147996 22686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:51:11.149255 22686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } }
I20260924 04:51:11.150234 22686 leader_election.cc:288] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [CANDIDATE]: Term 1 election: Requested vote from peers 3bc38459cf5847fda080bdbb9df217f9 (127.0.98.61:43123), f7fb5dcea52647b6b50f568dec2e27c9 (127.0.98.60:45223)
I20260924 04:51:11.151932 22837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c3815267d324788a0ab1dc0b2d55d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f7fb5dcea52647b6b50f568dec2e27c9"
I20260924 04:51:11.152184 22837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:51:11.152503 22761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c3815267d324788a0ab1dc0b2d55d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3bc38459cf5847fda080bdbb9df217f9"
I20260924 04:51:11.152714 22761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:51:11.153492 22837 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c3815267d324788a0ab1dc0b2d55d97 in term 1.
I20260924 04:51:11.153916 22761 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c3815267d324788a0ab1dc0b2d55d97 in term 1.
I20260924 04:51:11.154191 22661 leader_election.cc:302] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c3815267d324788a0ab1dc0b2d55d97, f7fb5dcea52647b6b50f568dec2e27c9; no voters:
W20260924 04:51:11.154239 22665 tablet.cc:2369] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:51:11.155027 22886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:51:11.155184 22886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [term 1 LEADER]: Becoming Leader. State: Replica: 6c3815267d324788a0ab1dc0b2d55d97, State: Running, Role: LEADER
I20260924 04:51:11.155553 22886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } }
I20260924 04:51:11.157469 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c3815267d324788a0ab1dc0b2d55d97. Latest consensus state: current_term: 1 leader_uuid: "6c3815267d324788a0ab1dc0b2d55d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } } }
I20260924 04:51:11.157644 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:11.157975 22719 master_runner.cc:263] No leader master found from master 3bc38459cf5847fda080bdbb9df217f9
I20260924 04:51:11.158039 22719 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 04:51:11.158195 22908 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:51:11.158278 22908 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:51:11.160547 22644 master_runner.cc:263] No leader master found from master 6c3815267d324788a0ab1dc0b2d55d97
I20260924 04:51:11.160600 22644 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:51:11.162540 22916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:51:11.165792 22916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:51:11.168285 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.158753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.0.98.62:36993)
0924 04:51:11.158862 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.158866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.159488 (+ 622us) server_negotiation.cc:407] Connection header received
0924 04:51:11.159697 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.159700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.159747 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.159797 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.160862 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.161551 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.162651 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.162951 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.163064 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.164818 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:11.164821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:11.164824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:11.164849 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:11.166402 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.166891 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.166895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.166897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.167007 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.167309 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.167312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.167314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.167523 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:11.167618 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:11.168011 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 04:51:11.168132 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:51:11.177804 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.161321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.0.98.60:45223)
0924 04:51:11.161418 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.161421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.168832 (+ 7411us) server_negotiation.cc:407] Connection header received
0924 04:51:11.169358 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.169362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.169418 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.169466 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.170152 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.170946 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.172096 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.172438 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.172512 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.174160 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:11.174164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:11.174167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:11.174194 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:11.175850 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.176324 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.176328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.176330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.176415 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.176730 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.176733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.176734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.176950 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:11.177051 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:11.177395 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 04:51:11.177521 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:51:11.183485 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:11.161969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.0.98.61:43123)
0924 04:51:11.162069 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.162072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.169270 (+ 7198us) server_negotiation.cc:407] Connection header received
0924 04:51:11.170264 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.170267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.170317 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.170366 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.171217 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.171975 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.178164 (+ 6189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:11.178476 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:11.178554 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:11.180071 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:11.180075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:11.180079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:11.180109 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:11.181583 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.182106 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.182110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.182111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.182196 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:11.182474 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:11.182477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:11.182479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:11.182727 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:11.182847 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:11.183218 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 04:51:11.183335 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:51:11.184165 22837 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9 [term 1 FOLLOWER]: Refusing update from remote peer 6c3815267d324788a0ab1dc0b2d55d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:51:11.184931 22886 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:51:11.190287 22870 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c3815267d324788a0ab1dc0b2d55d97. Latest consensus state: current_term: 1 leader_uuid: "6c3815267d324788a0ab1dc0b2d55d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } } }
I20260924 04:51:11.190423 22870 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:51:11.191838 22761 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9 [term 1 FOLLOWER]: Refusing update from remote peer 6c3815267d324788a0ab1dc0b2d55d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:51:11.192507 22913 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:51:11.196748 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c3815267d324788a0ab1dc0b2d55d97. Latest consensus state: current_term: 1 leader_uuid: "6c3815267d324788a0ab1dc0b2d55d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } } }
I20260924 04:51:11.196870 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:51:11.199774 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c3815267d324788a0ab1dc0b2d55d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } } }
I20260924 04:51:11.199889 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:11.199890 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c3815267d324788a0ab1dc0b2d55d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } } }
I20260924 04:51:11.199975 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:51:11.200246 22921 mvcc.cc:205] Tried to move back new op lower bound from 7332763529960370176 to 7332763529856782336. Current Snapshot: MvccSnapshot[applied={T|T < 7332763529960370176}]
I20260924 04:51:11.207182 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c3815267d324788a0ab1dc0b2d55d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } } }
I20260924 04:51:11.207309 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97 [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:11.208455 22870 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c3815267d324788a0ab1dc0b2d55d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3815267d324788a0ab1dc0b2d55d97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36993 } } peers { permanent_uuid: "3bc38459cf5847fda080bdbb9df217f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "f7fb5dcea52647b6b50f568dec2e27c9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45223 } } }
I20260924 04:51:11.208531 22922 mvcc.cc:205] Tried to move back new op lower bound from 7332763529960370176 to 7332763529856782336. Current Snapshot: MvccSnapshot[applied={T|T < 7332763529960370176}]
I20260924 04:51:11.208575 22870 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:51:11.211349 22924 mvcc.cc:205] Tried to move back new op lower bound from 7332763529960370176 to 7332763529856782336. Current Snapshot: MvccSnapshot[applied={T|T < 7332763529960370176}]
I20260924 04:51:11.212877 22916 catalog_manager.cc:1347] Generated new cluster ID: 3c89fd3764b4447487b83ef181d3f57e
I20260924 04:51:11.212949 22916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:51:11.238554 22916 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:51:11.239903 22916 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:51:11.258536 22916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c3815267d324788a0ab1dc0b2d55d97: Generated new TSK 0
I20260924 04:51:11.259546 22916 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:51:11.294698 22716 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:11.292094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.0.98.62:36993)
0924 04:51:11.292245 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.292250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.293570 (+ 1320us) server_negotiation.cc:407] Connection header received
0924 04:51:11.293786 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.293790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.293840 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.293892 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.294544 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49248: BlockingRecv error: recv got EOF from 127.0.0.1:49248 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 04:51:11.296162 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:11.292587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.0.98.60:45223)
0924 04:51:11.292734 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.292738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.295228 (+ 2490us) server_negotiation.cc:407] Connection header received
0924 04:51:11.295413 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.295416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.295464 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.295515 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.296033 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40916: BlockingRecv error: recv got EOF from 127.0.0.1:40916 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 04:51:11.297497 22793 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:11.292957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.0.98.61:43123)
0924 04:51:11.293088 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.293093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.296654 (+ 3561us) server_negotiation.cc:407] Connection header received
0924 04:51:11.296822 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.296825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.296874 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.296922 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.297384 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59888: BlockingRecv error: recv got EOF from 127.0.0.1:59888 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 04:51:11.302893 22793 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:11.301348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.0.98.61:43123)
0924 04:51:11.301464 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.301470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.301931 (+ 461us) server_negotiation.cc:407] Connection header received
0924 04:51:11.302140 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.302143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.302219 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.302271 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.302783 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59896: BlockingRecv error: recv got EOF from 127.0.0.1:59896 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 04:51:11.308388 22793 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:11.306734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.0.98.61:43123)
0924 04:51:11.306860 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.306864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.307411 (+ 547us) server_negotiation.cc:407] Connection header received
0924 04:51:11.307615 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.307618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.307688 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.307744 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.308279 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59902: BlockingRecv error: recv got EOF from 127.0.0.1:59902 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 04:51:11.314476 22793 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:11.312799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.0.98.61:43123)
0924 04:51:11.312915 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.312919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.313462 (+ 543us) server_negotiation.cc:407] Connection header received
0924 04:51:11.313691 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.313694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.313752 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.313808 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.314352 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59908: BlockingRecv error: recv got EOF from 127.0.0.1:59908 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 04:51:11.322744 22793 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:11.320938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.0.98.61:43123)
0924 04:51:11.321094 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.321099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.321764 (+ 665us) server_negotiation.cc:407] Connection header received
0924 04:51:11.321972 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.321976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.322030 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.322081 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.322618 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59914: BlockingRecv error: recv got EOF from 127.0.0.1:59914 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 04:51:11.335233 22793 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:11.333313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.0.98.61:43123)
0924 04:51:11.333440 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.333445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.334112 (+ 667us) server_negotiation.cc:407] Connection header received
0924 04:51:11.334336 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.334340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.334397 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.334452 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.335097 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59916: BlockingRecv error: recv got EOF from 127.0.0.1:59916 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 04:51:11.354707 22793 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:11.353486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.0.98.61:43123)
0924 04:51:11.353611 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.353615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.353876 (+ 261us) server_negotiation.cc:407] Connection header received
0924 04:51:11.354010 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.354013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.354063 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.354132 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.354589 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59920: BlockingRecv error: recv got EOF from 127.0.0.1:59920 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 04:51:11.389588 22793 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:11.388599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.0.98.61:43123)
0924 04:51:11.388774 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.388778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.388917 (+ 139us) server_negotiation.cc:407] Connection header received
0924 04:51:11.389001 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.389004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.389055 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.389136 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.389491 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59934: BlockingRecv error: recv got EOF from 127.0.0.1:59934 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 04:51:11.456884 22793 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:11.455817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.0.98.61:43123)
0924 04:51:11.455988 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.455992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.456086 (+ 94us) server_negotiation.cc:407] Connection header received
0924 04:51:11.456254 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.456257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.456312 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.456392 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.456768 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59948: BlockingRecv error: recv got EOF from 127.0.0.1:59948 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 04:51:11.588331 22793 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:11.587030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.0.98.61:43123)
0924 04:51:11.587170 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.587175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.587378 (+ 203us) server_negotiation.cc:407] Connection header received
0924 04:51:11.587534 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.587537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.587594 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.587758 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.588213 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59956: BlockingRecv error: recv got EOF from 127.0.0.1:59956 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 04:51:11.847460 22793 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:11.846378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.0.98.61:43123)
0924 04:51:11.846507 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:11.846512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:11.846581 (+ 69us) server_negotiation.cc:407] Connection header received
0924 04:51:11.846731 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:11.846733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:11.846783 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:11.846856 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:11.847343 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59958: BlockingRecv error: recv got EOF from 127.0.0.1:59958 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:51:12.143088 22882 catalog_manager.cc:1259] Loaded cluster ID: 3c89fd3764b4447487b83ef181d3f57e
I20260924 04:51:12.143190 22882 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9: loading cluster ID for follower catalog manager: success
I20260924 04:51:12.144821 22882 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9: acquiring CA information for follower catalog manager: success
I20260924 04:51:12.146008 22882 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7fb5dcea52647b6b50f568dec2e27c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:51:12.159714 22908 catalog_manager.cc:1259] Loaded cluster ID: 3c89fd3764b4447487b83ef181d3f57e
I20260924 04:51:12.159801 22908 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9: loading cluster ID for follower catalog manager: success
I20260924 04:51:12.161504 22908 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9: acquiring CA information for follower catalog manager: success
I20260924 04:51:12.162745 22908 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3bc38459cf5847fda080bdbb9df217f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:51:12.365314 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:12.361454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.0.98.61:43123)
0924 04:51:12.361704 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:12.361709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:12.361725 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:51:12.361875 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:12.361878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:12.362000 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:12.362096 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:51:12.362386 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:12.362930 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:12.363842 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:12.364062 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:12.364147 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:12.364482 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:51:12.364930 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:51:12.365045 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:12.365103 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 04:51:12.365162 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260924 04:51:12.372008 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:12.368348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40932 (local address 127.0.98.60:45223)
0924 04:51:12.368622 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:12.368627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:12.368641 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:51:12.368726 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:12.368728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:12.368840 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:12.368940 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:51:12.369180 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:12.369688 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:12.370520 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:12.370769 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:12.370859 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:12.371190 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:51:12.371625 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:51:12.371737 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:12.371803 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 04:51:12.371859 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260924 04:51:12.379105 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:12.375128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40942 (local address 127.0.98.60:45223)
0924 04:51:12.375237 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:12.375241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:12.375562 (+ 321us) server_negotiation.cc:407] Connection header received
0924 04:51:12.375706 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:12.375710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:12.375758 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:12.375820 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:51:12.376109 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:12.376708 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:12.377980 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:12.378303 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:12.378389 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:12.378633 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:51:12.378716 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:51:12.378777 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:12.378880 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 04:51:12.378930 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:51:12.379410 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:12.375007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49260 (local address 127.0.98.62:36993)
0924 04:51:12.375252 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:12.375256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:12.375609 (+ 353us) server_negotiation.cc:407] Connection header received
0924 04:51:12.375848 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:12.375851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:12.376050 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:12.376170 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:51:12.376396 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:12.377075 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:12.377911 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:12.378235 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:12.378332 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:12.378620 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:51:12.378996 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:51:12.379073 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:12.379211 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 04:51:12.379273 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260924 04:51:12.379626 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:12.375159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59978 (local address 127.0.98.61:43123)
0924 04:51:12.375352 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:12.375356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:12.375606 (+ 250us) server_negotiation.cc:407] Connection header received
0924 04:51:12.375667 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:12.375670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:12.375725 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:12.375793 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:51:12.376003 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:12.376763 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:12.378455 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:12.378750 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:12.378842 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:12.379199 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:51:12.379279 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:51:12.379351 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:12.379450 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 04:51:12.379502 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 04:51:12.385761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 22644
I20260924 04:51:12.391213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 22719
I20260924 04:51:12.396205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 22794
2026-09-24T04:51:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2992 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:51:12 dist-test-slave-hvcl krb5kdc[23011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:51:12 dist-test-slave-hvcl krb5kdc[23011](info): set up 2 sockets
Sep 24 04:51:12 dist-test-slave-hvcl krb5kdc[23011](info): commencing operation
krb5kdc: starting...
W20260924 04:51:13.347153 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/test-user.keytab.
Sep 24 04:51:13 dist-test-slave-hvcl krb5kdc[23011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:51:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:51:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:51:13.469764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41693
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41693,127.0.98.61:45097,127.0.98.60:39075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:51:13.578056 23024 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:51:13.578333 23024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:51:13.578356 23024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:51:13.578402 23024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:51:13.582352 23024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:51:13.582410 23024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:51:13.582428 23024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:51:13.582443 23024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:51:13.582458 23024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:51:13.587774 23024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41693,127.0.98.61:45097,127.0.98.60:39075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:51:13.588800 23024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:51:13.589818 23024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:51:13.598054 23030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:13.598071 23032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:13.598071 23029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:13.598112 23024 server_base.cc:1034] running on GCE node
I20260924 04:51:13.598822 23024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:51:13.599931 23024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:51:13.601085 23024 hybrid_clock.cc:648] HybridClock initialized: now 1790225473601066 us; error 35 us; skew 500 ppm
Sep 24 04:51:13 dist-test-slave-hvcl krb5kdc[23011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:51:13.603765 23024 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:51:13.604605 23024 webserver.cc:460] Webserver started at http://127.0.98.62:45977/ using document root <none> and password file <none>
I20260924 04:51:13.605132 23024 fs_manager.cc:359] Metadata directory not provided
I20260924 04:51:13.605188 23024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:51:13.605346 23024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:51:13.607385 23024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/data/instance:
uuid: "563b7c7a5c2b46c8b570ecaad370956a"
format_stamp: "Formatted at 2026-09-24 04:51:13 on dist-test-slave-hvcl"
server_key: "6dea587932fe82224e4b92d295c83a59"
server_key_iv: "ebb2b7ac32ff984663875649fe85cf5a"
server_key_version: "encryptionkey@0"
I20260924 04:51:13.607885 23024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/wal/instance:
uuid: "563b7c7a5c2b46c8b570ecaad370956a"
format_stamp: "Formatted at 2026-09-24 04:51:13 on dist-test-slave-hvcl"
server_key: "6dea587932fe82224e4b92d295c83a59"
server_key_iv: "ebb2b7ac32ff984663875649fe85cf5a"
server_key_version: "encryptionkey@0"
I20260924 04:51:13.611570 23024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:51:13.614152 23039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:51:13.615518 23024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:51:13.615658 23024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/wal
uuid: "563b7c7a5c2b46c8b570ecaad370956a"
format_stamp: "Formatted at 2026-09-24 04:51:13 on dist-test-slave-hvcl"
server_key: "6dea587932fe82224e4b92d295c83a59"
server_key_iv: "ebb2b7ac32ff984663875649fe85cf5a"
server_key_version: "encryptionkey@0"
I20260924 04:51:13.615774 23024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:51:13.637997 23024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:51:13.641252 23024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:51:13.641433 23024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:51:13.648304 23024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41693
I20260924 04:51:13.648566 23091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41693 every 8 connection(s)
I20260924 04:51:13.649317 23024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:51:13.652275 23092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:51:13.655902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 23024
I20260924 04:51:13.656013 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-0/wal/instance
I20260924 04:51:13.656306 392 external_mini_cluster.cc:1428] Setting key 47c0725318d4a8086461b8f8bfe21073
I20260924 04:51:13.656032 23092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 04:51:13 dist-test-slave-hvcl krb5kdc[23011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:51:13.673357 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.658626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.0.98.62:41693)
0924 04:51:13.659072 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:13.659090 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:13.659384 (+ 294us) server_negotiation.cc:407] Connection header received
0924 04:51:13.660461 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:13.660467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:13.660724 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:13.660902 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:13.661986 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.662873 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.664350 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.664897 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.665025 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:13.668689 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:13.668696 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:13.668707 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:13.668767 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:13.670307 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:13.670850 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:13.670855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:13.670856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:13.670962 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:13.671307 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:13.671310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:13.671311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:13.671743 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:13.671848 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:13.672283 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 04:51:13.672504 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":113,"threads_started":1}
I20260924 04:51:13.673359 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.658627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41693 (local address 127.0.0.1:47190)
0924 04:51:13.659244 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.659282 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:13.660108 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:13.660941 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:13.660946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:13.661449 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:13.661810 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:13.661830 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.663018 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.663022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:13.664191 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:13.664202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.665036 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.665039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:13.665132 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:13.665775 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:13.665791 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:13.668528 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:13.670429 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:13.670439 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:13.670450 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:13.670746 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:13.671067 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:13.671070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:13.671072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:13.671213 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:13.671856 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:13.671863 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:13.672036 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 04:51:13.672226 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 04:51:13.672576 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":29,"spinlock_wait_cycles":42752,"thread_start_us":183,"threads_started":1}
I20260924 04:51:13.675979 23092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:51:13.676671 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.676435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45097 (local address 127.0.0.1:55096)
0924 04:51:13.676537 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.676610 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 04:51:13.676906 23041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45097: connect: Connection refused (error 111)
I20260924 04:51:13.677783 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.677535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45097 (local address 127.0.0.1:55112)
0924 04:51:13.677653 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.677709 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:51:13.677920 23092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45097: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45097: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:51:13.686153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45097
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41693,127.0.98.61:45097,127.0.98.60:39075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:51:13.727152 23092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } attempt: 1
I20260924 04:51:13.728013 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.727733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45097 (local address 127.0.0.1:55120)
0924 04:51:13.727877 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.727937 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 04:51:13.728680 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.728478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45097 (local address 127.0.0.1:55122)
0924 04:51:13.728579 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.728622 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:51:13.728786 23092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45097: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45097: connect: Connection refused (error 111)
W20260924 04:51:13.794029 23099 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:51:13.794358 23099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:51:13.794394 23099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:51:13.794461 23099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:51:13.796954 23092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } attempt: 2
I20260924 04:51:13.797825 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.797512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45097 (local address 127.0.0.1:55128)
0924 04:51:13.797681 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.797747 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 04:51:13.798578 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.798355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45097 (local address 127.0.0.1:55134)
0924 04:51:13.798472 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.798519 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 04:51:13.798758 23092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45097: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45097: connect: Connection refused (error 111)
W20260924 04:51:13.798776 23099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:51:13.798826 23099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:51:13.798866 23099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:51:13.798897 23099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:51:13.798925 23099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:51:13.804425 23099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41693,127.0.98.61:45097,127.0.98.60:39075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:51:13.805569 23099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:51:13.806609 23099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:51:13.815261 23105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:13.815261 23104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:13.815392 23099 server_base.cc:1034] running on GCE node
W20260924 04:51:13.815286 23107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:13.816198 23099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:51:13.817349 23099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:51:13.818518 23099 hybrid_clock.cc:648] HybridClock initialized: now 1790225473818498 us; error 34 us; skew 500 ppm
Sep 24 04:51:13 dist-test-slave-hvcl krb5kdc[23011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:51:13.821501 23099 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:51:13.822367 23099 webserver.cc:460] Webserver started at http://127.0.98.61:35501/ using document root <none> and password file <none>
I20260924 04:51:13.822889 23099 fs_manager.cc:359] Metadata directory not provided
I20260924 04:51:13.822947 23099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:51:13.823156 23099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:51:13.825178 23099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/data/instance:
uuid: "6096753a3e494f8d957ec331dd2b819f"
format_stamp: "Formatted at 2026-09-24 04:51:13 on dist-test-slave-hvcl"
server_key: "044f593fe94663114b96c72580f6633f"
server_key_iv: "4d73065edba53dc316440deabe7fa808"
server_key_version: "encryptionkey@0"
I20260924 04:51:13.825693 23099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/wal/instance:
uuid: "6096753a3e494f8d957ec331dd2b819f"
format_stamp: "Formatted at 2026-09-24 04:51:13 on dist-test-slave-hvcl"
server_key: "044f593fe94663114b96c72580f6633f"
server_key_iv: "4d73065edba53dc316440deabe7fa808"
server_key_version: "encryptionkey@0"
I20260924 04:51:13.829367 23099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 04:51:13.831823 23114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:51:13.833092 23099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:51:13.833192 23099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/wal
uuid: "6096753a3e494f8d957ec331dd2b819f"
format_stamp: "Formatted at 2026-09-24 04:51:13 on dist-test-slave-hvcl"
server_key: "044f593fe94663114b96c72580f6633f"
server_key_iv: "4d73065edba53dc316440deabe7fa808"
server_key_version: "encryptionkey@0"
I20260924 04:51:13.833281 23099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:51:13.859462 23099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:51:13.862236 23099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:51:13.862392 23099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:51:13.869035 23099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45097
I20260924 04:51:13.869302 23166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45097 every 8 connection(s)
I20260924 04:51:13.869984 23099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:51:13.872469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 23099
I20260924 04:51:13.872586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-1/wal/instance
I20260924 04:51:13.872918 392 external_mini_cluster.cc:1428] Setting key 2e657315c36c493b61bced0faadc4915
I20260924 04:51:13.873023 23167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:51:13.877071 23167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 04:51:13 dist-test-slave-hvcl krb5kdc[23011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:51:13.889935 23092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } attempt: 3
I20260924 04:51:13.890645 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.879042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.0.98.62:41693)
0924 04:51:13.879163 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:13.879167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:13.880274 (+ 1107us) server_negotiation.cc:407] Connection header received
0924 04:51:13.881234 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:13.881238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:13.881304 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:13.881380 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:13.882562 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.883156 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.884325 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.884568 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.884661 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:13.887173 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:13.887176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:13.887178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:13.887204 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:13.888542 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:13.889083 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:13.889087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:13.889088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:13.889164 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:13.889474 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:13.889477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:13.889479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:13.889662 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:13.889747 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:13.890354 (+ 607us) server_negotiation.cc:299] Negotiation successful
0924 04:51:13.890488 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:51:13.891652 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.879569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41693 (local address 127.0.0.1:47196)
0924 04:51:13.880116 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.880159 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:13.881020 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:13.881560 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:13.881567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:13.881928 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:13.882415 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:13.882430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.883287 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.883290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:13.884175 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:13.884187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.884719 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.884722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:13.884923 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:13.885583 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:13.885604 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:13.887002 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:13.888648 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:13.888654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:13.888672 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:13.888985 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:13.889244 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:13.889247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:13.889249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:13.889389 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:13.889760 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:13.889767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:13.890084 (+ 317us) client_negotiation.cc:769] Sending connection context
0924 04:51:13.890296 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 04:51:13.890628 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":159,"threads_started":1}
I20260924 04:51:13.893249 23167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:51:13 dist-test-slave-hvcl krb5kdc[23011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:51:13.898161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
/tmp/dist-test-taskI0qsqU/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39075
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41693,127.0.98.61:45097,127.0.98.60:39075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:51:13 dist-test-slave-hvcl krb5kdc[23011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:51:13.901698 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.890377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45097 (local address 127.0.0.1:55144)
0924 04:51:13.890510 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.890524 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:13.890597 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:13.891432 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:13.891435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:13.891636 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:13.891775 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:13.891783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.892926 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.892929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:13.893666 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:13.893675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.894392 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.894396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:13.894504 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:13.895046 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:13.895060 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:13.896549 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:13.900041 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:13.900044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:13.900047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:13.900295 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:13.900563 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:13.900565 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:13.900567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:13.900653 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:13.901176 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:13.901179 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:13.901285 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 04:51:13.901386 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 04:51:13.901528 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 04:51:13.901754 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.890577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.0.98.61:45097)
0924 04:51:13.891053 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:13.891065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:13.891087 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:51:13.891124 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:13.891128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:13.891301 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:13.891442 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:13.891883 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.892820 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.893815 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.894276 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.894384 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:13.896677 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:13.896685 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:13.896697 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:13.896724 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:13.899940 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:13.900392 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:13.900399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:13.900401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:13.900481 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:13.900749 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:13.900752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:13.900753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:13.901067 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:13.901193 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:13.901467 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 04:51:13.901607 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":99,"threads_started":1}
I20260924 04:51:13.902202 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.893788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45097 (local address 127.0.0.1:55152)
0924 04:51:13.893899 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.893911 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:13.893973 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:13.894433 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:13.894436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:13.894589 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:13.894669 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:13.894674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.895702 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.895705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:13.896483 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:13.896493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.897007 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.897010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:13.897078 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:13.897389 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:13.897402 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:13.898567 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:13.900929 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:13.900932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:13.900934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:13.901101 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:13.901358 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:13.901361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:13.901363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:13.901442 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:13.901794 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:13.901797 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:13.901866 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 04:51:13.901936 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:51:13.902056 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 04:51:13.902709 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.893872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.0.98.61:45097)
0924 04:51:13.894244 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:13.894253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:13.894265 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:51:13.894297 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:13.894300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:13.894334 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:13.894415 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:13.894774 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.895602 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.896584 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:13.896906 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:13.896995 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:13.898697 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:13.898700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:13.898703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:13.898730 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:13.900838 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:13.901188 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:13.901191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:13.901193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:13.901277 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:13.901526 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:13.901529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:13.901530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:13.901689 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:13.902373 (+ 684us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:13.902445 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 04:51:13.902550 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":155,"threads_started":1}
I20260924 04:51:13.903944 23167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:51:13.904119 23092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:51:13.904709 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.904403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:40922)
0924 04:51:13.904563 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.904642 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 04:51:13.904760 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.904520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:40936)
0924 04:51:13.904630 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.904683 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 04:51:13.904938 23115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
I20260924 04:51:13.905558 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.905329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:40942)
0924 04:51:13.905451 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.905496 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 04:51:13.905680 23092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39075: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
I20260924 04:51:13.905994 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.905774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:40946)
0924 04:51:13.905875 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.905928 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 04:51:13.906172 23167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39075: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
I20260924 04:51:13.936854 23092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } attempt: 1
I20260924 04:51:13.937785 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.937447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:40960)
0924 04:51:13.937607 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.937682 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 04:51:13.938490 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.938281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:40962)
0924 04:51:13.938376 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.938427 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:51:13.938612 23092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39075: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
I20260924 04:51:13.955438 23167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } attempt: 1
I20260924 04:51:13.956329 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.955994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:40964)
0924 04:51:13.956155 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.956230 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 04:51:13.957100 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:13.956879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:40968)
0924 04:51:13.956983 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:13.957036 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 04:51:13.957281 23167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39075: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
W20260924 04:51:14.010519 23174 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:51:14.010812 23174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:51:14.010836 23174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:51:14.010892 23174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:51:14.013774 23092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } attempt: 2
I20260924 04:51:14.014575 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.014268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:40970)
0924 04:51:14.014418 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:14.014486 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 04:51:14.015256 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.015056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:40980)
0924 04:51:14.015147 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:14.015195 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 04:51:14.015302 23174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:51:14.015349 23174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:51:14.015375 23092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39075: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
W20260924 04:51:14.015417 23174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:51:14.015456 23174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:51:14.015484 23174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:51:14.021087 23174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41693,127.0.98.61:45097,127.0.98.60:39075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 04:01:11 UTC on ccycloud.quasar-mywcyt.root.comops.site
build id 4669299
I20260924 04:51:14.022133 23174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:51:14.023222 23174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:51:14.025450 23167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } attempt: 2
I20260924 04:51:14.026250 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.025933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:40990)
0924 04:51:14.026088 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:14.026158 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 04:51:14.026964 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.026737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:40996)
0924 04:51:14.026847 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:14.026901 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 04:51:14.027128 23167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39075: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39075: connect: Connection refused (error 111)
W20260924 04:51:14.031773 23180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:14.031829 23181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:51:14.031833 23183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:51:14.031998 23174 server_base.cc:1034] running on GCE node
I20260924 04:51:14.032546 23174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:51:14.033675 23174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:51:14.034857 23174 hybrid_clock.cc:648] HybridClock initialized: now 1790225474034829 us; error 38 us; skew 500 ppm
Sep 24 04:51:14 dist-test-slave-hvcl krb5kdc[23011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:51:14.037869 23174 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:51:14.038772 23174 webserver.cc:460] Webserver started at http://127.0.98.60:40251/ using document root <none> and password file <none>
I20260924 04:51:14.039358 23174 fs_manager.cc:359] Metadata directory not provided
I20260924 04:51:14.039422 23174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:51:14.039620 23174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:51:14.041723 23174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/data/instance:
uuid: "dfb467bccfe045f6876728c414353f62"
format_stamp: "Formatted at 2026-09-24 04:51:14 on dist-test-slave-hvcl"
server_key: "9f5388733d015a9d978985d03fb90447"
server_key_iv: "e05172c25af9b3091ab7b09c1d8f221c"
server_key_version: "encryptionkey@0"
I20260924 04:51:14.042268 23174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/wal/instance:
uuid: "dfb467bccfe045f6876728c414353f62"
format_stamp: "Formatted at 2026-09-24 04:51:14 on dist-test-slave-hvcl"
server_key: "9f5388733d015a9d978985d03fb90447"
server_key_iv: "e05172c25af9b3091ab7b09c1d8f221c"
server_key_version: "encryptionkey@0"
I20260924 04:51:14.045991 23174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:51:14.048447 23190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:51:14.049657 23174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:51:14.049774 23174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/wal
uuid: "dfb467bccfe045f6876728c414353f62"
format_stamp: "Formatted at 2026-09-24 04:51:14 on dist-test-slave-hvcl"
server_key: "9f5388733d015a9d978985d03fb90447"
server_key_iv: "e05172c25af9b3091ab7b09c1d8f221c"
server_key_version: "encryptionkey@0"
I20260924 04:51:14.049876 23174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:51:14.090106 23174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:51:14.093508 23174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:51:14.093709 23174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:51:14.100476 23174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39075
I20260924 04:51:14.100800 23242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39075 every 8 connection(s)
I20260924 04:51:14.101657 23174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:51:14.104420 23243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:51:14.105371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu as pid 23174
I20260924 04:51:14.105479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI0qsqU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225194677594-392-0/minicluster-data/master-2/wal/instance
I20260924 04:51:14.105759 392 external_mini_cluster.cc:1428] Setting key b579a259172b70b7bda3affa15932e6d
I20260924 04:51:14.108713 23243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:51:14.114521 23092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } attempt: 3
Sep 24 04:51:14 dist-test-slave-hvcl krb5kdc[23011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:51:14 dist-test-slave-hvcl krb5kdc[23011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:51:14.118264 23167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } attempt: 3
Sep 24 04:51:14 dist-test-slave-hvcl krb5kdc[23011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:51:14.120128 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.110613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47206 (local address 127.0.98.62:41693)
0924 04:51:14.110748 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.110753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.111055 (+ 302us) server_negotiation.cc:407] Connection header received
0924 04:51:14.111177 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.111180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.111237 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.111307 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.111949 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.112497 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.113454 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.113792 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.113903 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.116656 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:14.116660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:14.116663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:14.116688 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:14.118079 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.118524 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.118528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.118530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.118610 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.118999 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.119003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.119005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.119223 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:14.119307 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:14.119837 (+ 530us) server_negotiation.cc:299] Negotiation successful
0924 04:51:14.119970 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:51:14.121368 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.110529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41693 (local address 127.0.0.1:47206)
0924 04:51:14.110912 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:14.110959 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:14.111094 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:14.111372 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:14.111377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:14.111720 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:14.111841 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:14.111846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.112601 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.112605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:14.113347 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:14.113356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.113933 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.113936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:14.114017 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.114524 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:14.114543 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:14.116505 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:14.118178 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:14.118187 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:14.118193 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:14.118438 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:14.118717 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:14.118720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:14.118722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:14.118877 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:14.119362 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:14.119368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:14.119522 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 04:51:14.120299 (+ 777us) client_negotiation.cc:240] Negotiation successful
0924 04:51:14.120553 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":3,"thread_start_us":119,"threads_started":1}
I20260924 04:51:14.121376 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.107129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.0.98.60:39075)
0924 04:51:14.107549 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.107569 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.107613 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:51:14.108461 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.108484 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.108920 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.109290 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.109931 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.111279 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.112182 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.112692 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.112844 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.115679 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:14.115686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:14.115699 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:14.115721 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:14.117466 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.117957 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.117964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.117971 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.118104 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.118717 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.118720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.118722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.119303 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:14.119499 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:14.120175 (+ 676us) server_negotiation.cc:299] Negotiation successful
0924 04:51:14.120635 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20260924 04:51:14.122788 23243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:51:14.124029 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.114891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:41006)
0924 04:51:14.115012 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:14.115025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:14.115099 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:14.115458 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:14.115461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:14.115612 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:14.115711 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:14.115718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.116698 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.116702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:14.117495 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:14.117505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.118019 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.118023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:14.118097 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.118497 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:14.118512 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:14.119852 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:14.122363 (+ 2511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:14.122366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:14.122368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:14.122603 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:14.123019 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:14.123022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:14.123023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:14.123111 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:14.123536 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:14.123540 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:14.123657 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 04:51:14.123751 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 04:51:14.123879 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 04:51:14.124112 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.114946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41006 (local address 127.0.98.60:39075)
0924 04:51:14.115278 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.115287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.115297 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:51:14.115322 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.115324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.115354 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.115424 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.115827 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.116572 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.117605 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.117931 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.118022 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.119992 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:14.119995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:14.119998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:14.120020 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:14.122236 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.122792 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.122796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.122797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.122891 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.123211 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.123214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.123216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.123414 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:14.123493 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:14.123839 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 04:51:14.123945 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":102,"threads_started":1}
Sep 24 04:51:14 dist-test-slave-hvcl krb5kdc[23011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:51:14.128087 23092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a: Bootstrap starting.
Sep 24 04:51:14 dist-test-slave-hvcl krb5kdc[23011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:51:14 dist-test-slave-hvcl krb5kdc[23011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:51:14.131055 23092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a: Neither blocks nor log segments found. Creating new log.
I20260924 04:51:14.132020 23092 log.cc:824] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:51:14.132628 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.123523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.0.98.61:45097)
0924 04:51:14.123630 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.123634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.124257 (+ 623us) server_negotiation.cc:407] Connection header received
0924 04:51:14.124353 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.124355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.124399 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.124475 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.124877 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.125578 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.126561 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.126918 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.127016 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.129121 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:14.129125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:14.129127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:14.129152 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:14.130696 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.131228 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.131235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.131237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.131327 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.131655 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.131659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.131661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.131947 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:14.132022 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:14.132333 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 04:51:14.132452 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":896}
I20260924 04:51:14.132628 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.119204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:41008)
0924 04:51:14.119492 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:14.119507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:14.119583 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:14.122725 (+ 3142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:14.122729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:14.122922 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:14.123082 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:14.123091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.124162 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.124166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:14.124827 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:14.124836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.125836 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.125839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:14.125925 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.126367 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:14.126381 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:14.128130 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:14.130732 (+ 2602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:14.130736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:14.130738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:14.130996 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:14.131503 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:14.131506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:14.131508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:14.131606 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:14.132234 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:14.132237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:14.132324 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 04:51:14.132395 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:51:14.132498 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260924 04:51:14.133136 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.123306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45097 (local address 127.0.0.1:55168)
0924 04:51:14.124195 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:14.124209 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:14.124286 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:14.124509 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:14.124512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:14.124677 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:14.124776 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:14.124783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.125686 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.125689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:14.126435 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:14.126446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.127157 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.127160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:14.127259 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.127665 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:14.127679 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:14.128974 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:14.130792 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:14.130796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:14.130797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:14.131036 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:14.131455 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:14.131458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:14.131460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:14.131541 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:14.132090 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:14.132092 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:14.132159 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 04:51:14.132849 (+ 690us) client_negotiation.cc:240] Negotiation successful
0924 04:51:14.132983 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826}
I20260924 04:51:14.133234 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.119712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.0.98.60:39075)
0924 04:51:14.122491 (+ 2779us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.122499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.122509 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:51:14.122538 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.122541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.122579 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.123194 (+ 615us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.123300 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.124031 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.125389 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.125712 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.128255 (+ 2543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.128356 (+ 101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:14.128359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:14.128361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:14.128379 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:14.130614 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.131328 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.131331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.131333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.131404 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.131707 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.131709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.131711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.132135 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:14.132911 (+ 776us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:14.132983 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 04:51:14.133072 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2717,"thread_start_us":91,"threads_started":1}
I20260924 04:51:14.133788 23243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:51:14.134282 23092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a: No bootstrap required, opened a new log
I20260924 04:51:14.135564 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.124317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47214 (local address 127.0.98.62:41693)
0924 04:51:14.124799 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.124803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.124995 (+ 192us) server_negotiation.cc:407] Connection header received
0924 04:51:14.125093 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.125095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.125136 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.125194 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.125882 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.126532 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.127576 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.127866 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.127932 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.130775 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:14.130778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:14.130781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:14.130804 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:14.132555 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.134123 (+ 1568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.134137 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.134139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.134222 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.134580 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.134583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.134585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.134787 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:14.134920 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:14.135306 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 04:51:14.135425 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413}
I20260924 04:51:14.136149 23167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f: Bootstrap starting.
I20260924 04:51:14.138150 23092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } }
I20260924 04:51:14.138340 23092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563b7c7a5c2b46c8b570ecaad370956a, State: Initialized, Role: FOLLOWER
I20260924 04:51:14.138661 23092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } }
I20260924 04:51:14.139073 23167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f: Neither blocks nor log segments found. Creating new log.
Sep 24 04:51:14 dist-test-slave-hvcl krb5kdc[23011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:51:14.139210 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } } }
I20260924 04:51:14.140002 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:51:14.140532 23092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:51:14.140692 23167 log.cc:824] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:51:14.142764 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.134249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39075 (local address 127.0.0.1:41020)
0924 04:51:14.134573 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 04:51:14.134584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:51:14.134670 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:51:14.134940 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:51:14.134943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:51:14.135118 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:51:14.135221 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:14.135227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.136166 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.136168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:14.136909 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:51:14.136919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.137510 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.137514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:51:14.137593 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.138031 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:51:14.138045 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:51:14.139523 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:51:14.141422 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:14.141426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:14.141428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:14.141661 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:14.141932 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:51:14.141935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:51:14.141937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:51:14.142008 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:51:14.142352 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:51:14.142355 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:51:14.142422 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 04:51:14.142494 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:51:14.142614 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286}
W20260924 04:51:14.143219 23045 tablet.cc:2369] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:51:14.143272 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.134437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.0.98.60:39075)
0924 04:51:14.134522 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.134526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.134623 (+ 97us) server_negotiation.cc:407] Connection header received
0924 04:51:14.134748 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.134751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.134788 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.134832 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.135342 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.136053 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.137023 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.137393 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.137496 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.139648 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:14.139652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:14.139654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:14.139675 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:14.141322 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.141751 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.141754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.141756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.141844 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.142087 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.142090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.142092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.142266 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:14.142943 (+ 677us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:14.143038 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 04:51:14.143145 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 04:51:14.143528 23167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f: No bootstrap required, opened a new log
I20260924 04:51:14.146224 23243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62: Bootstrap starting.
I20260924 04:51:14.146582 23167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } }
I20260924 04:51:14.146759 23167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6096753a3e494f8d957ec331dd2b819f, State: Initialized, Role: FOLLOWER
I20260924 04:51:14.147060 23167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } }
I20260924 04:51:14.148708 23167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:51:14.145279 23024 master_runner.cc:186] Error getting master registration for 127.0.98.61:45097: OK, instance_id { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" instance_seqno: 1790225473867227 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:51:14.149013 23024 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:51:14.149269 23243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62: Neither blocks nor log segments found. Creating new log.
I20260924 04:51:14.150159 23243 log.cc:824] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:51:14 dist-test-slave-hvcl krb5kdc[23011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260924 04:51:14.152428 23263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:51:14.152493 23263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 04:51:14.156215 23277 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:51:14.156288 23277 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:51:14.155598 23099 master_runner.cc:186] Error getting master registration for 127.0.98.60:39075: OK, instance_id { permanent_uuid: "dfb467bccfe045f6876728c414353f62" instance_seqno: 1790225474098560 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:51:14.158675 23099 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:51:14.158802 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.140187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.0.98.61:45097)
0924 04:51:14.140304 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.140308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.140322 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:51:14.140471 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.140475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.140525 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.140601 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.142187 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.142909 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.148302 (+ 5393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.148611 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.148686 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.151865 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:14.151869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:14.151872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:14.151900 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:14.153519 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.154160 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.154164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.154166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.154255 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.154567 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.154570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.154572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.154789 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:14.154956 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:14.155365 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 04:51:14.155496 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:51:14.159106 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } } }
I20260924 04:51:14.159229 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:51:14.162173 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:51:14.163395 23066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "563b7c7a5c2b46c8b570ecaad370956a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47214
I20260924 04:51:14.163642 23066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 04:51:14.163759 23066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:51:14.165025 23066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } }
I20260924 04:51:14.165469 23243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62: No bootstrap required, opened a new log
I20260924 04:51:14.166040 23066 leader_election.cc:288] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [CANDIDATE]: Term 1 election: Requested vote from peers 6096753a3e494f8d957ec331dd2b819f (127.0.98.61:45097), dfb467bccfe045f6876728c414353f62 (127.0.98.60:39075)
I20260924 04:51:14.166958 23141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "563b7c7a5c2b46c8b570ecaad370956a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6096753a3e494f8d957ec331dd2b819f"
I20260924 04:51:14.167234 23141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:51:14.168448 23141 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563b7c7a5c2b46c8b570ecaad370956a in term 1.
I20260924 04:51:14.168931 23041 leader_election.cc:302] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563b7c7a5c2b46c8b570ecaad370956a, 6096753a3e494f8d957ec331dd2b819f; no voters:
I20260924 04:51:14.169238 23251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:51:14.169477 23251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [term 1 LEADER]: Becoming Leader. State: Replica: 563b7c7a5c2b46c8b570ecaad370956a, State: Running, Role: LEADER
I20260924 04:51:14.169804 23251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } }
I20260924 04:51:14.169739 23243 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } }
I20260924 04:51:14.169936 23243 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb467bccfe045f6876728c414353f62, State: Initialized, Role: FOLLOWER
I20260924 04:51:14.170192 23243 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } }
I20260924 04:51:14.171387 23243 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:51:14.171882 23217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "563b7c7a5c2b46c8b570ecaad370956a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dfb467bccfe045f6876728c414353f62"
I20260924 04:51:14.172109 23217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:51:14.173393 23217 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 563b7c7a5c2b46c8b570ecaad370956a in term 1.
I20260924 04:51:14.174507 23283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 563b7c7a5c2b46c8b570ecaad370956a. Latest consensus state: current_term: 1 leader_uuid: "563b7c7a5c2b46c8b570ecaad370956a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } } }
I20260924 04:51:14.174701 23283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:14.179100 23284 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } } }
I20260924 04:51:14.179224 23284 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 04:51:14.180038 23296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:51:14.180107 23296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:51:14.180409 23297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:51:14.181279 23297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:51:14.200053 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.184025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.0.98.62:41693)
0924 04:51:14.184136 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.184140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.185880 (+ 1740us) server_negotiation.cc:407] Connection header received
0924 04:51:14.185919 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.185922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.185964 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.186028 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.187069 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.187849 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.189750 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.190086 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.190168 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.194384 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:14.194389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:14.194392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:14.194427 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:14.196136 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.197627 (+ 1491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.197632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.197633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.197732 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.199055 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.199059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.199061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.199291 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:14.199405 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:14.199759 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 04:51:14.199884 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:51:14.201781 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.183355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.0.98.60:39075)
0924 04:51:14.183472 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.183476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.186046 (+ 2570us) server_negotiation.cc:407] Connection header received
0924 04:51:14.186323 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.186327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.186372 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.186419 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.187297 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.188054 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.189181 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.189528 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.189597 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.191416 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:14.191420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:14.191423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:14.191451 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:14.193085 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.200439 (+ 7354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.200443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.200446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.200546 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.200844 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.200848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.200849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.201060 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:14.201178 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:14.201508 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 04:51:14.201626 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:51:14.206261 23217 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62 [term 1 FOLLOWER]: Refusing update from remote peer 563b7c7a5c2b46c8b570ecaad370956a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:51:14.207193 23141 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f [term 1 FOLLOWER]: Refusing update from remote peer 563b7c7a5c2b46c8b570ecaad370956a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:51:14.207831 23283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:51:14.208683 23283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:51:14.214728 23284 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 563b7c7a5c2b46c8b570ecaad370956a. Latest consensus state: current_term: 1 leader_uuid: "563b7c7a5c2b46c8b570ecaad370956a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } } }
I20260924 04:51:14.214910 23284 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:51:14.217113 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 563b7c7a5c2b46c8b570ecaad370956a. Latest consensus state: current_term: 1 leader_uuid: "563b7c7a5c2b46c8b570ecaad370956a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } } }
I20260924 04:51:14.217252 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:51:14.218163 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:14.183606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55188 (local address 127.0.98.61:45097)
0924 04:51:14.183703 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.183707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.185537 (+ 1830us) server_negotiation.cc:407] Connection header received
0924 04:51:14.185660 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.185663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.185714 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.185794 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.191971 (+ 6177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.193106 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.196619 (+ 3513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:14.196978 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:14.197054 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:14.201578 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:51:14.201582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:51:14.201585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:51:14.201614 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:51:14.204249 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.204708 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.204711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.204713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.204812 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:51:14.205084 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:51:14.205087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:51:14.205089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:51:14.205306 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:51:14.205406 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:14.217855 (+ 12449us) server_negotiation.cc:299] Negotiation successful
0924 04:51:14.218005 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:51:14.231899 23283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "563b7c7a5c2b46c8b570ecaad370956a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } } }
I20260924 04:51:14.232060 23283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:14.232168 23302 mvcc.cc:205] Tried to move back new op lower bound from 7332763542340464640 to 7332763542202822656. Current Snapshot: MvccSnapshot[applied={T|T < 7332763542340464640}]
I20260924 04:51:14.233316 23284 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "563b7c7a5c2b46c8b570ecaad370956a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } } }
I20260924 04:51:14.233443 23284 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:51:14.235110 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "563b7c7a5c2b46c8b570ecaad370956a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } } }
I20260924 04:51:14.235255 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a [sys.catalog]: This master's current role is: LEADER
I20260924 04:51:14.238426 23303 mvcc.cc:205] Tried to move back new op lower bound from 7332763542340464640 to 7332763542202822656. Current Snapshot: MvccSnapshot[applied={T|T < 7332763542340464640}]
I20260924 04:51:14.238884 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "563b7c7a5c2b46c8b570ecaad370956a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "563b7c7a5c2b46c8b570ecaad370956a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41693 } } peers { permanent_uuid: "6096753a3e494f8d957ec331dd2b819f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45097 } } peers { permanent_uuid: "dfb467bccfe045f6876728c414353f62" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39075 } } }
I20260924 04:51:14.239045 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:51:14.238983 23305 mvcc.cc:205] Tried to move back new op lower bound from 7332763542340464640 to 7332763542202822656. Current Snapshot: MvccSnapshot[applied={T|T < 7332763542340464640}]
I20260924 04:51:14.239691 23297 catalog_manager.cc:1347] Generated new cluster ID: 2f0ffc78341b4b989cb300b7dec4ada7
I20260924 04:51:14.239748 23297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:51:14.279673 23297 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:51:14.281014 23297 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:51:14.298840 23297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 563b7c7a5c2b46c8b570ecaad370956a: Generated new TSK 0
I20260924 04:51:14.299726 23297 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:51:14.336131 23095 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:14.334083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47232 (local address 127.0.98.62:41693)
0924 04:51:14.334223 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.334228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.334848 (+ 620us) server_negotiation.cc:407] Connection header received
0924 04:51:14.335122 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.335126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.335188 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.335247 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.335976 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47232: BlockingRecv error: recv got EOF from 127.0.0.1:47232 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 04:51:14.338955 23248 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:14.337103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41038 (local address 127.0.98.60:39075)
0924 04:51:14.337242 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.337247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.337947 (+ 700us) server_negotiation.cc:407] Connection header received
0924 04:51:14.338139 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.338144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.338199 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.338253 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.338824 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41038: BlockingRecv error: recv got EOF from 127.0.0.1:41038 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 04:51:14.340830 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:14.337502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55190 (local address 127.0.98.61:45097)
0924 04:51:14.337617 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.337622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.339789 (+ 2167us) server_negotiation.cc:407] Connection header received
0924 04:51:14.339974 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.339978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.340053 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.340111 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.340664 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55190: BlockingRecv error: recv got EOF from 127.0.0.1:55190 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 04:51:14.345772 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:14.344152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.0.98.61:45097)
0924 04:51:14.344263 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.344267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.344721 (+ 454us) server_negotiation.cc:407] Connection header received
0924 04:51:14.344969 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.344972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.345054 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.345113 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.345646 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55202: BlockingRecv error: recv got EOF from 127.0.0.1:55202 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 04:51:14.351380 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:14.349653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.0.98.61:45097)
0924 04:51:14.349780 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.349785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.350339 (+ 554us) server_negotiation.cc:407] Connection header received
0924 04:51:14.350548 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.350551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.350612 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.350668 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.351264 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55218: BlockingRecv error: recv got EOF from 127.0.0.1:55218 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 04:51:14.357012 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:14.355837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.0.98.61:45097)
0924 04:51:14.355939 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.355943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.356193 (+ 250us) server_negotiation.cc:407] Connection header received
0924 04:51:14.356291 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.356294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.356357 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.356445 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.356899 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55230: BlockingRecv error: recv got EOF from 127.0.0.1:55230 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 04:51:14.364189 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:14.362818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.0.98.61:45097)
0924 04:51:14.363006 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.363010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.363279 (+ 269us) server_negotiation.cc:407] Connection header received
0924 04:51:14.363420 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.363424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.363480 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.363563 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.364065 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55246: BlockingRecv error: recv got EOF from 127.0.0.1:55246 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260924 04:51:14.376137 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:14.374818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.0.98.61:45097)
0924 04:51:14.374934 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.374939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.375253 (+ 314us) server_negotiation.cc:407] Connection header received
0924 04:51:14.375383 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.375387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.375448 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.375531 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.376013 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55260: BlockingRecv error: recv got EOF from 127.0.0.1:55260 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 04:51:14.395516 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:14.394385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.0.98.61:45097)
0924 04:51:14.394515 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.394518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.394667 (+ 149us) server_negotiation.cc:407] Connection header received
0924 04:51:14.394832 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.394835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.394883 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.394958 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.395395 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55262: BlockingRecv error: recv got EOF from 127.0.0.1:55262 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 04:51:14.431083 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:14.429625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.0.98.61:45097)
0924 04:51:14.429818 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.429823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.429982 (+ 159us) server_negotiation.cc:407] Connection header received
0924 04:51:14.430279 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.430282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.430334 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.430406 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.430930 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55278: BlockingRecv error: recv got EOF from 127.0.0.1:55278 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260924 04:51:14.498363 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:14.497175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.0.98.61:45097)
0924 04:51:14.497368 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.497372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.497470 (+ 98us) server_negotiation.cc:407] Connection header received
0924 04:51:14.497622 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.497625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.497684 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.497766 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.498240 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55292: BlockingRecv error: recv got EOF from 127.0.0.1:55292 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260924 04:51:14.629590 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:14.628480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.0.98.61:45097)
0924 04:51:14.628627 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.628633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.628745 (+ 112us) server_negotiation.cc:407] Connection header received
0924 04:51:14.628911 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.628913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.628960 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.629028 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.629488 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55308: BlockingRecv error: recv got EOF from 127.0.0.1:55308 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 04:51:14.889003 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:51:14.887803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.0.98.61:45097)
0924 04:51:14.887948 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:14.887951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:14.888143 (+ 192us) server_negotiation.cc:407] Connection header received
0924 04:51:14.888258 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:14.888261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:14.888313 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:14.888391 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:51:14.888901 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55316: BlockingRecv error: recv got EOF from 127.0.0.1:55316 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:51:15.157512 23277 catalog_manager.cc:1259] Loaded cluster ID: 2f0ffc78341b4b989cb300b7dec4ada7
I20260924 04:51:15.157593 23277 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f: loading cluster ID for follower catalog manager: success
I20260924 04:51:15.159258 23277 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f: acquiring CA information for follower catalog manager: success
I20260924 04:51:15.160388 23277 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6096753a3e494f8d957ec331dd2b819f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:51:15.181392 23296 catalog_manager.cc:1259] Loaded cluster ID: 2f0ffc78341b4b989cb300b7dec4ada7
I20260924 04:51:15.181483 23296 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62: loading cluster ID for follower catalog manager: success
I20260924 04:51:15.183264 23296 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62: acquiring CA information for follower catalog manager: success
I20260924 04:51:15.184496 23296 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dfb467bccfe045f6876728c414353f62: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:51:15.407051 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:15.403234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.0.98.61:45097)
0924 04:51:15.403449 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:15.403452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:15.403502 (+ 50us) server_negotiation.cc:407] Connection header received
0924 04:51:15.403628 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:15.403631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:15.403749 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:15.403851 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:51:15.404075 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:15.404615 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:15.405542 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:15.405807 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:15.405884 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:15.406179 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:51:15.406600 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:51:15.406690 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:15.406805 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 04:51:15.406854 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":67,"threads_started":1}
I20260924 04:51:15.413583 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:15.409934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.0.98.60:39075)
0924 04:51:15.410167 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:15.410170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:15.410184 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:51:15.410261 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:15.410262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:15.410372 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:15.410464 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:51:15.410609 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:15.411220 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:15.412043 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:15.412289 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:15.412379 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:15.412739 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:51:15.413181 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:51:15.413290 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:15.413360 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 04:51:15.413420 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260924 04:51:15.420773 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:15.417218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.0.98.60:39075)
0924 04:51:15.417326 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:15.417329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:15.417339 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:51:15.417381 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:15.417383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:15.417426 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:15.417498 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:51:15.417709 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:15.418473 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:15.419636 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:15.419938 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:15.420041 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:15.420307 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:51:15.420382 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:51:15.420442 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:15.420568 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 04:51:15.420629 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:51:15.421245 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:15.416730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.0.98.61:45097)
0924 04:51:15.416858 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:15.416862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:15.417462 (+ 600us) server_negotiation.cc:407] Connection header received
0924 04:51:15.417557 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:15.417560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:15.417612 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:15.417678 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:51:15.417994 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:15.418678 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:15.419764 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:15.420155 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:15.420242 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:15.420552 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:51:15.420628 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:51:15.420686 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:15.421059 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 04:51:15.421114 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:51:15.421993 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:51:15.417076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.0.98.62:41693)
0924 04:51:15.417300 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 04:51:15.417304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:51:15.417321 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:51:15.417817 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:51:15.417821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:51:15.418000 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:51:15.418104 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:51:15.418344 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:15.419129 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:15.420240 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:51:15.420598 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:51:15.420702 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:51:15.421109 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:51:15.421478 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:51:15.421558 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:51:15.421827 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 04:51:15.421885 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":60,"threads_started":1}
I20260924 04:51:15.424983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 23024
I20260924 04:51:15.430543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 23099
I20260924 04:51:15.435573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI0qsqU/build/debug/bin/kudu with pid 23174
2026-09-24T04:51:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3039 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6031 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (280749 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST